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Project management has become a key competence for most organisations in 
the public and private sectors. Driven by recent business trends such as fewer 
management layers, greater flexibility, increasing geographical distribution 
and more project-based work, project management has grown beyond its 
roots in the construction, engineering and aerospace industries to transform 
the service, financial, computer, and general management sectors. In fact, a 
Fortune article rated project management as the number one career choice at 
the beginning of the 21st century. Yet many organizations have struggled in 
applying the traditional models of project management to their new projects 
in the global environment. 

Project management offers a framework to help organisations to trans- 
form their mainstream operations and service performance. It is viewed as a 
way of organising for the future. Moreover, in an increasingly busy, stressful, 
and uncertain world it has become necessary to manage several projects suc- 
cessfully at the same time. According to some estimates the world annually 
spends well over $10 trillion (US) on projects. In the UK alone, more than 
£250 billion is spent on projects every year. Up to half of these projects fail! 
A major ingredient in the build-up leading to failure is often cited as the lack 
of adequate project management knowledge and experience. Some organiza- 
tions have responded to this situation by trying to improve the understanding 
and capability of their managers and employees who are introduced to pro- 
jects, as well as their experienced project managers in an attempt to enhance 
their competence and capability in this area. 

Routledge Frontiers in Project Management provides short, state of play, guides 
to the main aspects of the new emerging applications including: maturity 
models, agile projects, extreme projects, six sigma and projects, human fac- 
tors and leadership in projects, project governance, value management, vir- 
tual teams, project benefits. 


The Front-end of Large Public Projects 
Paradoxes and Ways Ahead 
Edited by Terry M. Williams, Knut Samset and Gro Holst Volden 


For more information about this series, please visit: https://www.routledge. 
com/Routledge-Frontiers-in-Project-Management/book-series/R FPM 


The Front-end of Large 
Public Projects 
Paradoxes and Ways Ahead 


Edited by 

Terry M. Williams, 
Knut Samset and 
Gro Holst Volden 


2 ¥ Routledge 
2 Taylor & Francis Group 


LONDON AND NEW YORK 


First published 2022 
by Routledge 
4 Park Square, Milton Park, Abingdon, Oxon OX14 4RN 


and by Routledge 
605 Third Avenue, New York, NY 10158 


Routledge is an imprint of the Taylor & Francis Group, an informa business 


© 2022 selection and editorial matter, Terry M. Williams, Knut 
Samset and Gro Holst Volden; individual chapters, the contributors 


The right of Terry M. Williams, Knut Samset and Gro Holst Volden to 
be identified as the authors of the editorial material, and of the authors 
for their individual chapters, has been asserted in accordance with 
sections 77 and 78 of the Copyright, Designs and Patents Act 1988. 


The Open Access version of this book, available at www. 
taylorfrancis.com, has been made available under a Creative 
Commons Attribution-Non Commercial-No Derivatives 4.0 
license. 


Trademark notice: Product or corporate names may be trademarks 
or registered trademarks, and are used only for identification and 
explanation without intent to infringe. 


British Library Cataloguing-in-Publication Data 
A catalogue record for this book is available from the British Library 


Library of Congress Cataloging-in-Publication Data 
A catalog record has been requested for this book 


ISBN: 978-1-032-18972-7 (hbk) 
ISBN: 978-1-032-18973-4 (pbk) 
ISBN: 978-1-003-25717-2 (ebk) 


DOT: 10.4324/9781003257172 


Typeset in Bembo 
by codeMantra 


Contents 


List of contributors vil 


1 Introduction 1 
GRO HOLST VOLDEN AND KNUT SAMSET 


2 Project success 10 
TERRY M. WILLIAMS 


3 The logic of the project front-end 43 
OFER ZWIKAEL AND ALICIA GILCHRIST 


4 Undertaking the project front-end 69 
MONIQUE AUBRY AND SERGHEI FLORICEL 


5 Estimation 99 
RICHARD J. KIRKHAM 


6 Incentives and politics: the perverse incentives 
paradox: root cause of many other paradoxes; the case 
of the Dutch Betuweroute 130 
BERT VAN WEE 


7 Closing the loop: ex ante and ex post evaluation in 
order to learn from mistakes and successes 158 
KNUT SAMSET AND GRO HOLST VOLDEN 


vi Contents 


8 Conclusions 191 
TERRY M. WILLIAMS 


Appendix 205 


Knut Samset and Gro Holst Volden (2016). Front-end 
definition of projects: Ten paradoxes and some reflections regarding 
project management and project governance. Re-printed from 
International Journal of Project Management, vol. 34 no. 2 
pp. 297-313. 


Index 239 


Contributors 


Monique Aubry is honorary professor in the Department of Management 
at ESG UQAM in Montréal, Canada. Now retired from full professorship, 
she pursues her research on two main topics: the development of meg- 
aprojects and organising for projects. The results of her work have been 
published in major project management journals. She founded the Lab for 
Transfer on Innovative Practices in Project Context at ESG UQAM, now 
being at the head of the executive board. In 2020, she was awarded the 
PMI Research Achievement Award for her entire career, and in 2012, she 
received the IPMA Research Award for her research on Project Manage- 
ment Offices. Over the last few years, she acts as external adviser on the 
management of public projects for several ministries and public organisa- 
tions in Quebec, Canada. 


Serghei Floricel is professor in the Department of Management at Univer- 
sity of Quebec in Montreal (UQAM), where he teaches project feasibility. 
His research focuses on the planning and organising of complex projects 
and on innovation processes. He has published, among others, in Interna- 
tional Journal of Project Management, Project Management Journal, R&D Man- 
agement, and Research-Technology Management. He is the lead author of two 
monographs published by the PMI: Increasing Project Flexibility and Refining 
the Knowledge Production Plan, and co-author of The Strategic Management of 
Large Engineering Projects (MIT Press, 2001). He was research director for 
the Managing Innovation in the New Economy (MINE) program, and PI 
for five other research projects. He holds degrees from the Technical Uni- 
versity of Civil Engineering of Bucharest, Romania and from UQAM. 


Alicia Gilchrist is a post-doctoral fellow of project management at the Aus- 
tralian National University’s Research School of Management. Alicia’s 
research interests are in benefits management, defence capability devel- 
opment, social alignment, organisational psychology and stakeholder rela- 
tionships. Her research focuses on project investment benefits realisation 
in defence. Alicia has contributed to a wide range of projects including 
projects examining leadership behaviours in defence, business-IT align- 
ment in the healthcare industry, and management accountants’ professional 
identity. Alicia’s research has appeared in refereed journals including the 


viii Contributors 


International Journal of Project Management and Communications of the ACM. 
She has degrees from the University of Canberra and Queensland Univer- 
sity of Technology. 


Richard J. Kirkham is a reader in civil engineering at The University 
of Manchester. He is an active researcher in project studies, particularly 
focusing on risk management in whole-life appraisal of buildings/infra- 
structure. Richard’s research in whole-life costing for infrastructure (in- 
corporated into ISO15686-5) continues through government advisory, 
particularly in major government projects, recognised with an ESRC se- 
condment to the Cabinet Office working in the IPA’s Portfolio Insight 
Team. Richard is CI in the ESRC-funded “Project X”, a major invest- 
ment seeking to improve the evidence base surrounding government’s 
major project/programme delivery. He was PI for two UK Government 
Cabinet Office-funded projects examining risk in government transfor- 
mation. Richard leads the “reliability and resilience” research theme in the 
Thomas Ashton Institute for Risk and Regulatory Research — a collabo- 
ration of the University of Manchester and The Health and Safety Execu- 
tive (Science Division). Richard also leads the “project delivery” research 
theme in the interdisciplinary Manchester Urban Institute, where he is 
currently leading a study on benefits realisation in nationally significant 
infrastructure projects for national highways. 


Bert van Wee is professor in transport policy at Delft University of Tech- 
nology, and scientific director of TRAIL research school. His research 
interests are in transport policy relevant issues such as environmental im- 
pacts, accessibility, evaluation of policies, plans, and infrastructure pro- 
jects, land-use and transport interaction, transport and ethics, and the role 
of technology in the transport system and related policy relevant effects. 
He is on editorial boards of various journals, for example, JAPA, Transport 
Reviews and Transport Policy, and a member of several international net- 
works. In 2014, he received the Association of American Geographers’ 
Edward L. Ullman Award for outstanding commitment and contribution 
to transport geography. In 2020, he received the Professor of Excellence 
Award of Delft University of Technology. He has degrees from Utrecht 
University and the University of Amsterdam. 


Ofer Zwikael is one of Australia’s most highly awarded project manage- 
ment scholars. His four books and over 250 peer-reviewed papers have 
been recognised through research awards from the Academy of Manage- 
ment, British Academy of Management, Project Management Institute, 
International Project Management Association, Emerald and the Austral- 
ian Institute of Project Management. He is the director of the Research 
School of Management at the Australian National University. His research 
on project benefits management has been funded by major organisations, 
including the Australian Department of Defence and the Project Manage- 
ment Institute. He has held leadership roles including associate dean and 
associate editor for the International Journal of Project Management. 


1 Introduction 
Gro Holst Volden and Knut Samset 


1.1 Paradoxes in front-end management 


The traditional focus of the project management community has, by and 
large, been restricted to what is termed ‘the iron triangle’ of cost, time and 
scope (Morris 2013). The iron triangle is an example of reductionist thinking 
where project performance is reduced to the ‘simple’ measures related to pro- 
ject implementation only. In recent years, many authors have argued the need 
for a wider, strategic view on projects, as the purpose of projects is essentially 
to deliver benefits and create value for the funding entity, for users and/or 
for society at large (Morris 2013; Samset & Volden 2016; Williams & Samset 
2010; Zwikael & Smyrk 2012). The focus of this book is on large public pro- 
jects, where a broad societal perspective on project outcome is particularly 
relevant — large public projects being tools for policy development. 

In line with such a broad interpretation of project success, there is an in- 
creasing recognition of the strategic role of the front-end phase in shaping 
the success of projects. The front-end phase is here defined as the period from 
when the initial idea is conceived to when the final implementation decision 
is made, during which it is still possible to make major changes or terminate 
the initiative at an affordable cost. Williams et al. (2019) refer to a number of 
studies which argue the case for using more resources in the front-end phase 
in order to improve project and portfolio success. 

It is a paradox in itself that this crucial phase of the project lifecycle is not 
better understood. An extensive literature review on the front-end phase of 
projects found that the literature on front-end management is fairly sparse, 
and that this phase is still not well understood (Williams et al. 2019). For 
example, it is not clear who the key players are at this stage, and how man- 
agement competencies should be improved. There is not even consensus as to 
whether the front end is part of the project lifecycle, or a separate undertaking 
that precedes the project. What seems clear, though, is that those who initiate 
the project are most likely from outside the project management community. 
Initiators of public investment projects might be politicians, the responsible 
ministry or agency (governing organisation), user groups or other stakehold- 
ers at local level. 
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There is clearly a need to understand how projects materialise from some 
initial conceptual idea or consideration. Whether actively encouraged or un- 
expectedly apparent, all projects are the result of some form of ambition and 
consideration. The front-end phase can be seen as the result of two processes 
that run in parallel: the analytic and decision-making processes. Williams 
et al. (2019) note that two key terms in this phase are ‘strategy’ and ‘context’. 
They suggest that the greater the maturity of the governing organisation in 
dealing with projects, the more structured and well-defined the management 
of the front end is likely to be. But there is still a gap in the literature. Much 
work has been done regarding project management, as well as on strategy 
formation at the organisational level, but little on how these two come 
together — which is, obviously, during the front-end phase. 

Samset and Volden (2016) presented research findings based on longitudi- 
nal research on the front-end management of major public investment pro- 
jects in Norway. The authors argued that many challenges and weaknesses 
need to be overcome to achieve project success such as the absence of a real- 
istic goal or purpose, lack of competence among planners, hidden agendas, 
processes driven by needs other than those of society at large, unrealistic and 
inconsistent assumptions and how to secure essential planning data and ad- 
equate contract regimes. More importantly, there was a tendency to ignore 
the crucial assessment of problems, needs, opportunity space and the choice 
of conceptual solution to the problem at hand, and instead jump directly to 
more detailed, and often quantitative and data-intensive, analyses of only one 
specific preconceived or preferred conceptual solution. 

These challenges and weaknesses were framed by the authors as ten par- 
adoxes that overlap to a varying extent. Paradoxes are here understood as 
situations with a counter-intuitive result, at least in the broad societal per- 
spective. This paper was what initiated the collective work on this book. It 
is referred to as the ‘paradoxes paper’ throughout the book. The full paper is 
included as an appendix at the end of the book. 

In short, the paradoxes are: 


1 The success paradox: success is measured in operational terms 
only, rather than the wider, strategic perspective. Projects that are 
completed with considerable cost overrun and behind schedule gener- 
ate negative media attention and even public inquiries, irrespective of 
whether they are relevant and good value for money. By contrast, pro- 
jects may avoid negative attention if completed on budget, regardless of 
their strategic success. 

2 The paradox of the significance of front-end management: less 
resources are used up front to identify the best conceptual solu- 
tion (project governance) than to improve performance during 
implementation (project management). The choice of conceptual 
solution often originates in the mind of an individual, based on intuition 
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and experience, rather than systematic analysis of problems, needs, re- 
quirements, etc. By contrast, comprehensive planning and analysis is as- 
sociated with the project once the choice of concept is made. 

The paradox of early information overflow: decisions are con- 
founded by masses of detailed information rather than carefully 
selected facts and judgments to highlight the essential issues. The 
priority should be to establish an overall perspective based on a targeted 
search for information. Experience shows that large amounts of detailed 
data at an early stage may result in what is referred to as ‘analysis paral- 
ysis. Instead of opening up the opportunity space, it may, in fact, lock 
decisions into an initially preferred concept. 

The paradox of the unexplored opportunity space: the choice of 
conceptual solution is made without systematically scrutinising 
the opportunity space up front. There is much evidence to suggest 
that in many cases the chosen concept is not necessarily the most effective 
solution to the initiating problem. In many cases, the process started out 
with a predetermined solution, without exploring other options. This is 
referred to as path dependency. 

The paradox of strategic alignment: strategy and alignment of 
objectives are highlighted as essential, but in many cases the 
internal logic of causality and probability of realisation are er- 
roneous. Alignment of objectives is the exercise of defining the causal 
link from the project outputs to outcome and long-term benefits of the 
project. Unfortunately, this is not always done. Objectives are missing 
or unclear, and there may be design faults at different levels, such as too 
many, overly ambitious and even conflicting goals. 

The cost estimation paradox: effort is made to get the final 
cost estimate (the budget) right, while early cost estimates are 
treated superficially. The ‘real decision’ is made at an early stage, based 
on initial estimates that are often substantially underestimated. There is 
much to suggest that this may result in the approval of projects that oth- 
erwise should have been rejected at an early stage. 

The paradox of disregarded analyses of costs and benefits: de- 
tailed estimation of cost and benefits is commonly done up 
front, but disregarded by decision-makers. Substantial amounts of 
resources are devoted to cost-benefit analyses, especially for transport 
projects. However, the estimated value for money had no significant im- 
pact on the selection of projects in Norway. On the contrary, many un- 
profitable projects were realised. Obviously, decision-makers emphasise 
other aspects, but these are not included in the analyses. 

The paradox of ‘predict and provide’: the tendency is to choose 
a ‘predict-and-provide’ strategy rather than explore alternative 
solutions. A variant of Paradox 4 (‘opportunity space’) is that in the 
case of congestion problems, need is often defined narrowly as the need 
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to increase capacity. While excess demand for public services and infra- 
structure is to be expected when offered free-of-charge to citizens, in 
some cases, there may be goals for a different development. Project own- 
ers need to clarify the needs and goals that should apply to the project. 

9 The paradox of perverse incentives: availability of public fund- 
ing with no financial obligations for the beneficiaries may cause 
perverse incentives and result in counter-productive projects. 
Different actors may have vested interest in certain projects being chosen, 
with no incentive to opt for the most socially beneficial or cost-effective 
alternative. This may result, inter alia, in supersized projects, positively 
biased business cases and the selection of projects that turn out to be 
complete failures. 

10 The paradox of myopic decisions: long-term viability is the 
intention, but the planning horizon is too short, resulting in 
sub-optimal choices. The study of project appraisals shows that needs 
and benefits are often assessed in a short-sighted and static perspective; 
trends are extrapolated without discussing alternative scenarios; and sig- 
nificant risk factors, such as political risk, are not identified and dis- 
cussed. Such practice may lead to decisions that society will regret in the 
future. 


The overall picture is that there are certain recurring deficiencies in analytic 
as well as decision-making processes, and that the potential for improvement 
is considerable. In fact, the ‘paradoxes paper’ found that flaws in both pro- 
cesses may be correlated, and further that projects with many such deficien- 
cies in the front-end phase tend to end up being less relevant to society. 

In a subsequent doctoral thesis, Volden (2019) discussed possible explana- 
tions for the observed paradoxes. Planners and analysts, who are often engi- 
neers and economists, may be hesitant to question fundamental issues that can 
be considered part of what is conceived as the political sphere. We have all 
heard analysts say, “We produce analyses, not guesswork”, implying that they 
are more comfortable working with tangible measures and clearly defined 
tools and methods than with multidimensional and qualitative assessments 
of success criteria that may be unclear and even disputed. There may also be 
cognitive shortcomings to innovative thinking, to applying long-term per- 
spectives and to planners’ understanding of fundamental uncertainties. An- 
other quite likely explanation is that project initiators (who often commission 
the analyses) see it in their interest to explore only one specific conceptual 
alternative, and restrict the terms of reference accordingly. Or even worse, 
they do not endorse an early project appraisal at all. 

Perverse incentives can be found at different administrative levels in 
society, and may cause other paradoxes discussed in this book. We have 
seen this in Norway, in the case of roads, hospitals, universities, sporting 
events, etc., where the local administrative level has been a key promoter, 
often in collaboration with other stakeholder groups and even members 
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of parliament. This is a country where the local democracy stands strong, 
while at the same time local government is financially weak and dependent 
on the national government to finance local infrastructure. This may have 
given rise to serious problems with adverse incentives on the part of local 
initiators. 

From this previous work, some key improvement measures are highlighted: 


e The business case should be presented to decision-makers early enough 
to prevent premature lock-in to an unjustified concept. 

e Incentives for project initiators ought to be brought in line with society’s 
interests as much as possible. Adverse incentives relating to discretionary 
assessment and approval processes need to be dealt with. 

e Analyses should be transparent and overseen by independent experts. 


The funding entity (which, in the case of state-funded projects, is the gov- 
ernment on behalf of all tax payers) should put in place a set of processes, sys- 
tems and regulations up front, in order to ensure project success, strategically 
as well as tactically. This is referred to as project governance (Williams & 
Samset 2012), and is closely related to the topic of the present book. In fact, 
front-end paradoxes and project governance need to be understood and dis- 
cussed together. Project governance should potentially be essential to over- 
come the front-end paradoxes. However, in order for the project governance 
framework to be effective, we first need to fully understand the paradoxes 
and how they work in different contexts. 


1.2 Aim of the book and introduction to each chapter 


The ‘paradoxes paper’ was our first probe into the matter. The paper defined 
a set of paradoxes to highlight various deficiencies in the front-end phase. 
However, we did not provide a thorough explanation of the paradoxes, the 
relationship between them or how to overcome them. Further, the findings 
were mostly based on Norwegian experiences. With the present book, the 
intention has been to investigate front-end paradoxes further — from different 
angles and with experiences from different countries, with the aim to achieve 
a deeper — and, hopefully, more generic — understanding, and to identify ef- 
fective remedies or solutions. 

The authors are all major experts in the field of front-end management and 
project governance. The book consists of six main chapters and a concluding 
one, which are briefly introduced below. Readers will learn about front- 
end paradoxes in various case projects from the United Kingdom, Australia, 
Canada, the Netherlands and Norway. The chapters and cases vary in terms 
of context (country, sector, etc.), their theoretical approach and the type of 
paradoxes they focus on. Together, they cover all ten paradoxes, and further 
develop the ideas about paradoxical dilemmas in front-end management and 
governance. 
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1.2.1 Understanding project success 


Chapter 2 is written by Professor Terry M. Williams from the University of 
Hull, who is also director of the Risk Institute. 

This chapter sets the scene for readers by contemplating what is meant by 
‘project success’, both in tactical and strategic terms. The logic is that the 
strategic success criteria should be considered first, with more attention being 
devoted later to tactical criteria as the project gradually takes shape. However, 
according to Paradox 1, ‘the success paradox’, in practice, minds tend to be 
focused mostly on efficiency targets. Williams discusses how this is related to 
difficulties in understanding what ‘strategic project success’ actually is, being 
a multifaceted, often difficult to measure and possibly a contested term. The 
chapter was also inspired by the related Paradox 10, ‘the paradox of myopic 
decisions’, that is, that projects are assessed from a short-term perspective — 
people want to be able to decide immediately if a project has been successful 
or not, without taking the time to wait for the verdict of history. 

Chapter 2 offers advice as to which issues need to be considered when 
defining a project’s strategic success, and illustrates the effects of governance 
mechanisms, and various analytic tools and practices that may be helpful in 
this phase. The discussion is based on literature and examples of good practice 
from the United Kingdom and elsewhere. 


1.2.2 How to construct an effective front-end phase 


Chapter 3 is written by Professor Ofer Zwikael and Dr Alicia Gilchrist from 
the Australian National University. 

They discuss the essential logic of the front-end phase and how this phase 
should be designed. It is assumed in the literature that the front-end phase 
begins with an idea, which, in turn, is triggered by a problem or an oppor- 
tunity. Yet, there is not a simple answer to what it takes to come up with a 
good project idea. In practice, there is often pressure to ‘be seen to be doing 
something’ with the problem at hand, and a tendency to jump to the seem- 
ingly best solution, without exploring options. 

Chapter 3 is particularly inspired by Paradox 2, ‘the significance of front- 
end management’, Paradox 4, ‘opportunity space’, and the related Paradox 8, 
‘predict and provide’. The Australian Defence Force, with its rigorous and 
advanced front-end phase, is used as a case study throughout the chapter, and 
recommendations are offered based on experiences from this sector. 

This chapter may thus assist practitioners in constructing an effective 
front-end phase that will facilitate the achievement of strategic objectives. 
There are also implications for the literature in providing suggestions as to 
how common front-end paradoxes may be resolved. 


1.2.3 The front end as seen from a social practice perspective 


Chapter 4 is written by Professors Monique Aubry and Serghei Floricel from 
The University of Québec in Montréal (UQAM), Canada. 
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This chapter relies on the notion of ‘project representation’. A representa- 
tion is a perceptual, conceptual and social construction which uses words, 
signs and drawings to describe the project and its context. The authors apply 
a social practice perspective to shed light on paradoxes and other difficulties 
in the front-end phase. According to this perspective, the development of 
a project representation is not seen as a ‘best option waiting to be selected’, 
but as a fragile, temporary outcome of multiple efforts. Processes are always 
emergent and follow multiple logics. 

The authors discuss four trade-offs relevant to the development of project 
representations. These trade-offs are then used to suggest explanations for 
Paradox 2, ‘the significance of front-end management’, and Paradox 4, ‘the 
opportunity space’. 

This chapter advances our understanding of front-end dynamics, as a pro- 
cess situated in time and having its own temporal logics. The focus on rep- 
resentations from a practice perspective will help readers grasp why front-end 
activities are rarely a linear unfolding process. The authors use several empir- 
ical vignettes from projects currently being developed in Quebec to demon- 
strate their points throughout the chapter. 


1.2.4 Exploring the cost estimation paradox 


Chapter 5 is written by Dr Richard Kirkham from the University of Man- 
chester, United Kingdom. 

This chapter looks at the process of cost estimation in the early phases of 
projects. Early cost estimates are often inaccurate and unreliable, some of 
the reasons being optimism bias and other cognitive issues that come into 
play. Others are incomplete information and availability of data. We also see 
projects being approved with no clear and realistic scope or objectives — in 
which case it is quite common that cost estimation will be insufficient as 
well. 

The author discusses a series of interrelated problems and possible solutions 
from the perspective of major project delivery in the United Kingdom. The 
chapter is inspired by Paradox 6, ‘the cost estimation paradox’ — that is, the 
focus on getting the final cost estimate right, while treating earlier cost esti- 
mates superficially. The discussion also touches on other related paradoxes, 
including Paradox 7, ‘the paradox of disregarded analyses of costs and bene- 
fits’. It is noted that government projects are truly uncertain in the front-end 
phase, and that the naive desire for commitment to early, often deterministic, 
estimates is in itself a paradox. 

The chapter makes a significant contribution to understanding fundamen- 
tal difficulties relating to cost estimation at the front end of projects. 


1.2.5 Incentives and politics 


Chapter 6 is written by Professor Bert van Wee from Delft University of 
Technology in the Netherlands. 
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The main topic of this chapter is Paradox 9, the ‘perverse incentives’ par- 
adox. The discussion centres around a case project, the Betuweroute, a rail 
freight line connecting Rotterdam Harbour with the hinterland. This pro- 
ject had a very long front-end phase, with Rotterdam Harbour as the key 
promoter, in search of enhanced competitiveness relative to other harbours 
in France, Belgium and Germany. Other arguments were also raised in the 
process, not least environmental concerns, and notions that the project would 
be good for the economy. But in the end, the project experienced a large cost 
overrun, had negative effects on the environment and was not economically 
viable. 

The chapter reviews the front-end phase to explain what went wrong. 
It shows how the Betuweroute was a classic example of a project driven by 
perverse incentives, van Wee arguing that a fundamental problem is the way 
such projects are financed. 

The author also discusses how Paradox 9 is related to all of the other para- 
doxes and suggests that understanding the ‘perverse incentives’ paradox may 
be helpful in understanding, and hopefully overcoming the others as well. 


1.2.6 Learning from past mistakes and successes 


Chapter 7 is written by Professor Knut Samset and Dr Gro Holst Volden, the 
previous and current director of the Concept Research Programme at the 
Norwegian University of Science and Technology (NTNU). 

The authors argue that paradoxical dilemmas, such as those discussed in 
this book, could have been avoided if planners and managers were better at 
learning from experience. The striking absence of ex post evaluation of pub- 
lic projects was, in fact, discussed in the initial ‘paradoxes paper’ as the 11th 
paradox. 

Researchers at NTNU have, since 2012, conducted ex post evaluation of 
some of the largest public infrastructure projects in Norway, to determine 
their success ex post, tactically as well as strategically. The authors discuss 
some experiences with these evaluations and argue that there is much to learn 
across project types and sectors. Some are better at benefits management, 
others at cost control and still others at handling unintended consequences. 
They also discuss how ex post evaluation may contribute to learning and im- 
provement, depending on the results and recommendations being perceived, 
understood and used. 

The chapter ends with a discussion on how ex post evaluation can be help- 
ful in overcoming each of the ten paradoxes. 


1.3 Conclusion 


The concluding chapter is written by Professor Terry M. Williams, who pulls 
the threads together from the previous chapters. In so doing, he includes 
Paradox 3, ‘early information overflow’, which is implicitly discussed in all 
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chapters. Further, he discusses how the paradoxes are not ten independent 
entities, but are related causally. 

The chapter takes a cross-chapter view, and brings together thoughts on 
seven ideas that crop up in most, if not all, of the chapters: 


e the problem or need that triggered the project idea; 
e the jump to an early project solution; 

e — stakeholders and consultation; 

e information generation and flow in the project; 

e accountability for the results of the project; 

e and reflecting back on a project. 


The authors hope that this book will help decision-makers as well as the pub- 
lic to understand the decisions being made at the front end of major public 
projects, so as to avoid some of the behavioural traps, to make better decisions 
in paradoxical situations and to plan and deliver projects that actually provide 
our countries with the benefits they are supposed to, both efficiently and 
effectively. 
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2 Project success 
Terry M. Williams 


2.1 The nature of project success 


This book concerns the development, management and delivery of large 
public projects, with the acknowledgement that, often, this is not as suc- 
cessful as we would wish. Before we can investigate the issues around this, 
however, we need to consider what makes a ‘successful’ project? What are 
we trying to achieve by carrying out all these projects? This is important not 
just for an academic discussion of the projects but because any party trying to 
make a project ‘successful’ will be aiming for whatever is their definition of 
‘success’. This chapter will therefore first look at the academic background to 
this question, dividing the idea of success into strategic and tactical success. It 
will then look at the various paradoxes that accompany major public projects 
as criteria for success are developed. 

Project management was originally developed to achieve the successful 
delivery of large, complicated projects where the definition of what needed 
to be done, and why, was fairly clear. The so-called bodies of knowledge, 
the best known of which is the PMBOK (Project Management Institute 
2017), were developed with the accumulated knowledge from successfully 
achieving well-defined projects that were large, complicated and demand- 
ing. Barnes (1988) famously said (of construction projects) that “the client’s 
objectives are always a combination of the objectives for performance of the 
completed scheme, for achieving this performance within a named cost or 
budgetary limit and for getting the project into use by a target date” (p. 69). 
The threefold criterion of success — meeting cost, schedule and performance 
targets — has, in the last 50 years, been widely used as a standard project man- 
agement success criterion, often called the ‘iron triangle’. Project managers 
are commissioned to go and work on their projects, and come back with 
them delivered to the specified iron triangle targets. 

As projects in the real world have developed, certain problems have been 
encountered with this definition. Some projects deemed successful according 
to this criterion did not seem, on the face of it, to be successful. The Zwen- 
tendorf Nuclear Power Plant (EVN 2020) was the first commercial nuclear 
electric-generation plant, built in Austria. Construction began in April 1972 
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and was completed in four years; however, a referendum was held on 5 No- 
vember 1978, in which a slim majority voted against starting the reactor up, 
so it has never operated as a nuclear reactor. An “on-shore torpedo battery 
built in rock on the northern coast of Norway in 2004” — huge and complex, 
accommodating 150 military personnel — was “officially opened as planned 
and without cost overrun. However just one week later it was closed down 
by Parliamentary resolution” since the concept of permanent torpedo bat- 
teries was obsolete (Samset 2010, p. 13). On the other hand, projects such as 
the Sydney Opera House or the Scottish Parliament, famously over-budget 
and late, but producing iconic buildings, might be considered unsuccessful 
according to the ‘iron triangle’ definition, but are successful in other, perhaps 
more important ways. 

Projects are not set up simply to achieve the project itself — they are set up 
for a purpose. Morris, in much of his work (e.g. Morris 2009), shows how 
corporate and business strategy is implementation by the use of projects. This 
is particularly true in the domain of public projects, the subject of this book. 
Tony Meggs, then chief executive of the UK’s Infrastructure and Projects 
Authority (which oversees all UK major government projects), wrote in his 
blog that 


The vast majority of government policies are delivered through the 
implementation of a project or programme of some description. These 
projects and programmes span a wide range ... [but] have one thing in 
common: if the projects are not successfully implemented, then the pol- 
icy objectives are not delivered. 

(Meggs 2018) 


Clearly the definition of success therefore needed to broaden out to in- 
clude the underlying strategic aim of a project. Is it useful? Does it do 
what we set out to do? Over time, therefore, many authors have come to 
distinguish between what might be termed the tactical success (‘project 
management success’ or ‘efficiency’ success of a project: did it fulfil the 
immediate specification as set out at the start of the project?) and the stra- 
tegic success (‘project success’ or “effectiveness success’: did it provide the 
outcome and benefits envisaged?). This recognition of the twofold nature 
of the concept of project success is becoming widely recognised and will 
be used in this chapter. 

Even then, this idea of ‘strategic success’ is not necessarily well-defined, for 
a number of reasons, and we will look at six particular issues, all of which will 
be touched upon later in the chapter. 

First, major public projects have a long lifespan, so ‘success’ can be regarded 
with a shorter or longer-term view. Perhaps the most influential definition of 
project success looking specifically at this was developed through work with 
the U.S. Agency for International Development, then the United Nations, 
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and OECD (Samset 2010, Chapter 2). This characterised project success 
as having five dimensions, starting with the immediate project, working 
through its immediate benefits, and through to the wider and longer-term 
aspects (see Table 2.1). 

This definition has proved useful for looking at major public projects. 
Zwikael and Meredith (2020) came up with a similar, three-stage defini- 
tion, but focusing on different viewpoints: project management success, the 
performance of the project manager in achieving the project plan; project 
ownership success, the project owner’s performance in realising the business 
case; and project investment success, the investment performance of the pro- 
ject for the funder. 

One curious feature of taking a shorter or longer-term view is that stake- 
holders’ view of ‘project failure’ is not a simple inverse of their view of ‘pro- 
ject success’. Chipulu et al. (2019) found that stakeholders’ assessment of 
project ‘success’ appeared more focused on project effectiveness, but when 
assessing project ‘failure’, they appeared more focused on efficiency. A cur- 
sory reading of the newspapers reflects this in the public discourse: reports 
of ‘project failure’ often focus on projects running out of control in terms of 
budget and time, whereas reports of ‘project success’ rarely talk about budgets 
or timescales, but rather the project output (e.g. the building or system pro- 
duced). This is in the public view — discussions of, say, National Audit Office 
assessments in this chapter show a more balanced view. 

Particularly in public projects, there is a wide range of different stakehold- 
ers, all of whom will have quite different perceptions of what constitutes 
project success, so our second point is the need to recognise these. There 
is a plethora of literature on stakeholders, but it is, perhaps, particularly 
within public projects that the range of stakeholders and heterogeneity of 
their views on project success is so clear. Politicians, public opinion, local 
residents, business, regulators, NGOs — the list of influential stakeholders 
can be considerable. The literature also shows the importance of recognising 


Table 2.1 Successive success criteria (Samset 2010) 


The Project Short-Term 


1 Efficiency Was the project well managed? 

2 Effectiveness Were the goals achieved? 

3 Relevance How useful was the output to the 
organisation? 

4 Impact Was the goal appropriate to the 
organisation’s purpose? 

5 Sustainability Are the benefits sustainable in the 


longer term? 


Wider concerns Longer-term 
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and bringing together these views: a poor common understanding across the 
range of project stakeholders can impact upon benefit realisation (O’Leary 
2012) in any project. In complex infrastructure projects, Wahab (2011) shows 
the importance of reconciling perceptions of benefits across often disparate 
stakeholder groups during the design process. Having said that, a comprehen- 
sive literature survey in Davis (2014) shows little commonality between the 
definitions of success among senior management, project teams and project 
recipient stakeholders. We will look at some examples of stakeholder views 
in this chapter. 

Much of the literature covers the idea of comparing the costs of a project, 
and the benefits that accrue from that project — the simplest view being a 
straightforward ‘cost-benefit analysis’. For some straightforward projects, this 
might be quite appropriate, but, as our third point, for most major public 
projects, the different types of benefits (or disbenefits) that might result from 
a project will not be easily quantifiable. Even where a benefit may be meas- 
urable, it might be difficult to turn that metric into a financial figure. For 
this reason, in many domains, governments suggest standard financial values 
for particular measurable benefits — transportation departments, for example, 
will often give financial value to reducing journey times by x minutes, or 
even a value for loss of life. Williams et al. (2020a) describe how countries 
such as the UK, Australia, Canada and Norway, and bodies such as the EU 
have detailed rules for quantifying benefits, generally emanating from their 
finance ministries. 

The combination of disparate measures calculated in terms of finance raises 
a number of issues, such as the accounting conventions used, interest rates, 
how to evaluate through-the-life impact of a project and so on. Moreover, 
for important public projects, some of the benefits or disbenefits might be 
simply subjective and unmeasurable — such as ‘social cohesion’, ‘visual amen- 
ity’ or even ‘national security’. Here attempts to measure the effect, let alone 
monetise it, might have little prospect of giving helpful advice. However, the 
idea of ‘social impact bonds’ is a useful development where a desired outcome 
is clear and measurable, but not obviously monetisable, for example reducing 
recidivism (see UK Government 2017). But for many projects, these might 
be some of the most important aspects. It is here that the differing views 
between different stakeholders discussed above can particularly become an 
issue. We will explore some examples in this chapter. 

We need to decide where the ‘impact’ of a project finishes. Our fourth 
point is that often a project has little effect until it goes into an operational 
delivery phase, and it is only then that benefits can be ‘harvested’. This could 
be citizens using a system, or a piece of infrastructure. A road project might 
facilitate local development — but only if the local authority or local business 
takes up those opportunities. Sometimes, in itself, a project might not be 
providing a benefit, but enabling others to achieve a benefit — in this sense, 
the ‘success’ of projects will be dependent upon changes in the behaviour 
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of citizens, business, government agencies, civil servants or other relevant 
stakeholders. 

Fifth, projects in a typical management environment can often be said to 
be “complex, ambiguous, confusing phenomena wherein the idea of a single, 
clear goal is at odds with the reality” (Linehan & Kavanagh, 2006). We have 
already pointed to the multiplicity of stakeholders, who might hold different 
views on what constitutes project ‘success’. Also, we have pointed to the mul- 
tiplicity of different success criteria, some of which might be measurable on 
the same scale, particularly if they can be expressed in some (perhaps proxy) 
financial terms — many of which will be incommensurable, or perhaps even 
unquantifiable. A project may be aiming for a number of targets. Further- 
more, these are often not separate goals but a complex web of causally related 
factors. A simple example is shown in Williams (2016), which, for a small 
set of projects in a small company, shows how success factors contributing to 
project performance combine in complex interactions, demonstrating causal 
paths from root causes to different but related success criteria. Even for this 
small example, final project success criteria, including, as well as the ‘iron 
triangle’ parameters about the final product (defects on building handover 
and in use and life cycle performance), stakeholder satisfaction (customers, 
users, community and subcontractors), project management success (health 
and safety) and the production of a legacy rather than just a building — and the 
causal chains leading to these — were complex and interlinked. 

Finally, for public projects, the surrounding environment can be turbulent 
and changing. The conventional approach to managing projects assumes that 
a project is defined, and then carried out according to its original target and 
specification. “Project management’ is difficult to envisage with constantly 
changing targets. This has long been recognised for projects in general: 


The Cartesian clarity of inner structures clashes with the increasing 
porosity of projects to complex contexts that they seek to deny.... The 
risk, in short, is that the idealistic ‘island of order’ may suddenly turn into 
a more realistic, very classic, ‘iron cage’. 

(Malgrati & Damiani 2002) 


For public projects, this turbulence is especially noticeable. Political land- 
scapes change. Major projects, particularly military or infrastructure, can 
take many years, whereas election cycles might only be four or five years, 
with a new government having quite different goals. Even if the government 
stays the same, in the UK, strategic spending reviews, which define the ob- 
jectives and thus the scale and nature of public service investments, take place 
every two to five years. Public opinion can be very fickle, and can influence 
the political motivations behind a project. Sometimes requirements change 
because technology has moved on (e.g. greater use of driverless cars may have 
a significant impact on the benefits expected by some transport infrastruc- 
ture projects — but again this is subject to the vagaries of public acceptability). 
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Sometimes, initial assumptions are simply wrong as decision-makers model 
how the world might change over the course of a project. 

In these circumstances, the idea of specifying a set of well-defined project 
goals which remain constant is not practical. Cicmil et al. (2006, p. 679) 
contrast “traditional approaches based on rational, objective, and universal 
representations of the project with a phronetic [practical wisdom] analysis of 
the ambiguous, fragmented and political reality of project situations”. Chap- 
ter 4 discusses the conceptual implications of undertaking a project front-end 
and show the development, over time, of circumstances and project work. 
Indeed, one of the current authors has written of “project organizations, as 
imperfect and fragile representations that chase a shifting nexus of intractable 
human, social, technical, and material processes” (Floricel et al. 2016). 

Given this academic introduction to the idea of ‘project success’, this chap- 
ter will explore how these ideas actually turn out in practice in some major 
public projects, touching on many of the reasons why defining project success 
criteria is not clear-cut. 

The chapter will look at the various stages of a project. We first explore 
what strategic success means and how targets are developed, then consider 
tactical success, taking a look at how this all evolves during project execution; 
we then look at the issues of success definition and project assessment after 
the project. As we explore the examples of projects, we will be looking at 
the realities of public projects and the environments in which they are born, 
developed and executed. 


2.2 Strategic success in public projects 


This section will take these considerations and look at what ‘strategic’ ben- 
efits mean in major public sector projects — what do we want out of our 
public projects, how is this defined, and how do projects arise out of these 
considerations? 


2.2.1 What should happen 


As discussed in the previous section, the starting point is not the project, 
but the policy purpose set out by the government — as described in the Tony 
Meggs quote above (Meggs 2018). In the same blog, Meggs talks about the 
search for “a seamless flow and inter-connectivity between policy concep- 
tion, policy development, and policy delivery”, this last increasingly through 
the medium of the project, as the public sector becomes increasingly projec- 
tified (e.g. Godenhjelm et al. 2015, in the EU). So how does this work out 
in practice? 

In the UK (this author’s home country), each government department sets 
out a ‘single departmental plan’, in which the Department sets out objectives 
and how they will be achieved. We are shortly to look at a transport case- 
study, so as an example, the UK Department of Transport sets out its plan 
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as a public document (Department of Transport 2019) with six overarching 
objectives (supporting the creation of a stronger, cleaner, more productive 
economy; helping to connect people and places; balancing investment across 
the country; making journeys easier, and so on). Some of these objectives 
are easier to quantify than others — some being more contested than others, 
and we shall see some examples. These departmental plans are supposed to 
set the foundation for the department’s programme portfolio — its individual 
programmes and the desired outcomes from projects — and the project out- 
puts that should provide those outcomes. This is laid out in the UK’s ‘Green 
Book’ (HM Treasury 2020), the ‘bible’ for appraising and evaluating major 
UK projects. Of course, it is not practical that all projects are proactively 
prompted by the departmental strategic objectives — some will be initiated by 
practical events or political motivations — but this does give a basis by which 
we can see how projects fit into the overall strategy. This type of process is 
explored in more detail (from an Australian viewpoint) in Chapter 3. 

Practically, governments are gradually developing systems by which the 
outputs likely to accrue from projects are identified, quantified and linked 
to these strategic priorities. This is sometimes badged as “benefits manage- 
ment’. A major PMI study looked at these systems in eight countries/inter- 
governmental organisations (IGOs) and found developments in all but one. 
Indeed, all of the other seven countries had explicit discussion in their docu- 
mentation linking project and national/government departmental goals — so 
at least the methods espoused and encouraged by the governments recognise 
this link. Schemes differed because of the nature of the countries/IGOs. The 
World Bank could be more integrated and focused. The physical size and 
federal structures of Canada and the US possibly explain the limited man- 
datory federal direction: perhaps benefits are better determined at the state/ 
province/local level. Australian state jurisdictions similarly have autonomy. 
Norway has a centralised method, but its size allows some informality, since 
people in the profession often know each other. The UK has traditionally had 
a separation between policy and delivery (although this is now decreasing). 
The EU is not one state, but a collection of states, so some parts of the pro- 
cess are carried out at state level. Work in four of these countries is reported 
in Williams et al. (2020a), showing Benefits Management frameworks be- 
ing used throughout, sometimes tailored to particular sectors (the transport 
and civil infrastructure sectors seemed particularly advanced). Some of these 
were advisory, except where they were mandated for the specific purpose of 
preparing business cases for final approval. It was noticeable that as projects 
progressed from approval through execution, the focus on benefits declined, 
as we will discuss below. 

However, as discussed in Section 2.1 above, “identifying and quantifying 
benefits’ is too simplistic. There is a high degree of heterogeneity in public 
project benefits. Simple financial or economic benefits are more straight- 
forward to recognise. A starting point is a classification system for benefits, 
since public projects in particular are undertaken to achieve a wide range of 
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financial and social benefits; the PMI Benefits study found many of these in 
practice (financial/non-financial; direct/indirect; a UK quadrant system; a 
Canadian five-stream system), but it was not clear how well-used these were 
(again, unless mandated for project approval). However, when we seek to 
improve the lives of the citizens of a country, we are in territory that is sub- 
jective and contested. Identifying benefits is therefore a process that needs to 
engage a wide range of stakeholders — which we will discuss below. The PMI 
study also showed that while some saw stakeholder engagement as an essen- 
tial ingredient in benefits identification, for others it was more of a cosmetic 
process, as it was unclear whether it affected project decision-making. 

Methods for quantifying benefits — an important ingredient for making 
out a business case for a project — appeared in the PMI study to lack standard- 
isation. Methods, sophistication of the processes and the degree to which the 
different methods were mandated all varied widely between different parts of 
government, although these again seemed particularly well developed in the 
transportation sector. Many benefits of public projects are difficult to define, 
let alone to quantify, or monetise; certainly a complete financial measurement 
of expected benefits is not usually a sensible aim. Current government sys- 
tems seem unlikely to be sufficient to measure many of these different types 
of benefits. Not surprisingly, the PMI study showed that a strong emphasis 
was put on easy-to-measure benefits, and those clearly and unambiguously 
linked to departmental strategic benefits. However, government projects 
span many types of project for which the main benefits are not quantifiable 
or monetisable, and it is not yet clear how these should be incorporated into 
a coherent government decision-making process. 


2.2.2 An example: the A303 project 


An example shows some of the different types of benefits, and some of the 
stakeholders involved. Stonehenge is a 4,000-year-old monument in the 
south of the UK, consisting of a ring of standing stones, each around 13 feet 
high and weighing around 25 tons. It is an iconic symbol of ancient Britain, 
a UNESCO World Heritage Site, and attracts many thousands of visitors, 
particularly at pagan festival times of year such as the summer solstice. There 
is a major road from the main part of England towards the holiday destina- 
tions of the south-west passing near Stonehenge, the A303. This has just one 
lane in each direction, and has long been recognised as a traffic problem, 
exacerbated by sightseers within their cars. It is generally felt to be a road that 
does not work, either for drivers, or for local residents, nor for travellers and 
holidaymakers. 

So there is a clearly recognised road-transportation problem. But equally 
clearly, this is not matter of a simple road upgrade. The nature of the World 
Heritage Site makes this a sensitive project, with many from across the UK 
seeing the site as part of their essential cultural heritage. The local villages, 
communities and groups also have strong views about the amenity and travel 
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Figure 2.1 Benefits of the ‘A303 project’. A summary of a map provided by Highways 
England (private correspondence with the author). 


around the locality. Moreover, it is a sensitive environmental area in terms of 
biodiversity, wildlife populations and movements (including a very rare UK 
bird, the stone curlew), air quality and noise. 

After much consultation and options analysis by Highways England (the 
agency responsible to the Department of Transport), a scheme including a 
2-mile tunnel to remove traffic on the A303 from the Stonehenge landscape 
was finally approved by the UK Secretary of State on 12 November 2020. 
More details can be found in their booklet (Highways England 2019). 

As can be seen from the description above, the benefits of this project and 
the criteria by which its success will be judged are wide and heterogeneous — 
indeed, the priority attributed to each is expected to evolve at different points 
of the project lifecycle. A sophisticated analysis was carried out by Highways 
England to identify, and where possible start to quantify these criteria. As 
well as identifying benefits, this will enable a robust scheme evaluation plan 
to consider the impacts of the scheme beyond its traditional transport and 
safety benefits. 

Figure 2.1 gives a map showing a much simplified version of this analysis, 
displaying the diversity of benefits. Here we can see, in the innermost part of 
the map, the fundamental Highways England departmental priorities leading 
to six diverse domains of benefit. Consideration of these domains leads to 
12 more specific areas in which those benefits will be realised. Each of these 
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areas is specified by multiple specific goals, each of which needs to be meas- 
ured: some might be straightforward to measure, such as road reliability or 
travel times; some will require customer surveys, such as enjoyment of the 
World Heritage Site (WHS); some might require considerable thought to 
measure, such as community cohesion. However, having this map accepted 
as part of the scheme gives an important basis to considering what the scheme 
is there for, and how successful it is. 


2.2.3 Some conceptual issues 


While the A303 project is a fine example of good practice in defining and 
starting to metricise benefits, there are a number of conceptual problems in 
this area which come up, as well as the problems that arise because we are 
dealing with individuals, stakeholders, companies and politics. Some of these 
conceptual issues are straightforward to contemplate, although that does not 
make the questions any easier to answer. 

One problem noted in Section 2.1 comes when the organisation respon- 
sible for executing the project is not the same as the organisation responsible 
for realising or ‘harvesting’ the benefits from the project. We will re-visit this 
issue, and the problems of accountability this raises in practice in Sections 
2.4/2.5. 

Another conceptual issue comes when we look at the set of different ‘ben- 
efits’. Generally these are considered individually and then put into a list. 
However, it is clear that often they are interlinked, and achievement of one 
will help (or hinder) achievement of the others. You only need to look at 
Figure 2.1 to see some interlinkages. Samset and Volden’s ‘Paradoxes’ paper 
notes an analysis of 17 Norwegian projects: 


A project strategy will always be a hierarchy of goals that are interlinked 
in cause-and-effect chains that illustrate the ambition levels for a project, 
as well as their realism. Objectives were analysed in terms of their inter- 
nal causality, and ambition. 

(p. 305) 


This helps to show both the interrelationship and also sometimes the distance 
between the project and the mooted benefit. Perhaps the most well-known 
structured method to bring these relationships out is the World Bank’s 
Logframe methodology; their Results Framework (Roberts & Khattri 2012) 
develops causal links from strategic objectives to project outcomes. 

A further issue was noted in Section 2.1: when would be an appropriate 
time to establish expected benefits? Benefits and disbenefits during the pro- 
ject period can be assessed during and at the end of that period. However, 
what about (to take the A303 example) economic activity, or health and 
wellbeing? Some of these might not be known for some considerable time 
after the project — others might have an immediate increase but then decline 
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back into life-as-usual (perhaps ‘the increased use of non-motorised trans- 
port’ mechanisms). There might be immediate requirements to assess the 
‘success’ of the project, but time needed to properly assess the longer-term 
benefits (this also plays into Samset and Volden’s Paradox 10, ‘the paradox of 
myopic decisions’, as discussed below). 

Another danger in assessing the benefit of a project is that it can ignore 
the wider portfolio of the government department. Programmes and projects 
rarely sit on their own, but contribute to the overall portfolio of programme 
activity in a Department, as stated in the Green Book discussed above. The 
UK has been building two aircraft carriers as a major element of their mil- 
itary defence. When the UK National Audit Office (2020b) reviewed the 
project, it found that the two carriers had been built, jets to go on the carriers 
received to schedule, and most of the surrounding infrastructure completed. 
However, an aircraft carrier does not act in isolation, and the report stated 
that the Ministry of Defence was, 


... making slower progress in developing the crucial supporting activities 
that are needed to make full use of a carrier strike group, such as .... In 
addition, it has not established a clear view on the future cost of enhanc- 
ing, operating and supporting Carrier Strike, which creates the risk of 
future affordability pressures. The Department will not achieve value for 
money from its investment to date unless it ... ensures cross-command 
coherence and collaboration to develop the full capabilities of Carrier 
Strike. 

National Audit Office (2020b, p. 11) 


We cannot evaluate the usefulness of an individual project without consider- 
ing its place in the portfolio of the Department’s programmes. 


2.2.4 A more fundamental conceptual issue 


But there is a more fundamental conceptual issue, which is that often a ‘ben- 
efit’ is not a well-established, black-and-white concept. The meaning of a 
benefit can be variable, and it can change over time. Impact can be multiple 
and equivocal, since it is valued in different (and often conflicting) ways. A 
continuation of the PMI study (Williams et al. 2020b) looked at case studies 
of three UK public projects, to consider the meanings of ‘benefit’, benefit 
changes, the effects of changes and tools for capturing change: the A303 
project above, transformation in the Department for Work and Pensions, and 
Digital Health. This led to a number of recommendations to capture the 
sometimes elusive nature of ‘benefits’, including: defining processes to define 
‘benefits’ terms; communicating with stakeholders in terms to which they 
can relate, particularly for societal benefits; developing tools that recognise 
the impossibility of capturing a ‘true’ permanent benefit and create a com- 
municative space for discussion; processes to recognise changes to benefits; 
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the use of narratives as a useful means of expressing benefits; and avoiding 
over-reliance on quantifiable benefits. The work for the A303 project, in 
particular, showed the wide variety of ways in which stakeholders benefit, 
the wide definition of benefits, the communicating of benefits (see below), 
and also the change over time of the benefits, as understanding of what can be 
achieved evolved, together with the perception of benefits, while the ‘core’ 
benefits remained fairly stable. 


2.2.5 Estimating 


While there is not time to explore this in detail, it needs to be noted that 
the identification and quantification of likely project outcomes is undertaken 
by individuals, with their natural biases. Flyvbjerg (notably Flyvbjerg et al. 
2003) has written extensively about the tendency towards ‘optimism bias’ 
and also the less savoury deliberate ‘strategic misrepresentation’ or ‘gaming’: 
over-estimation of project benefits (and under-estimation of costs) for the 
sake of achieving project approval. This will be covered more in Chapters 5 
and 6, and in the next sub-section, as we look at setting tactical success crite- 
ria. However, it is worth noting that in the study of many countries’ systems 
by Williams et al. (2020a), all governments’ guidance recognised the issue, 
practitioners saying they considered the tendency when putting project pro- 
posals together. It seemed that only the UK required a specific approach to 
quantifying optimism bias, the Green Book requiring a contingency to be 
placed on estimates, calculated using Reference Class Forecasting. 

While there has been considerable analysis of project databases to try to 
detect ‘optimism bias’, one clear problem with looking at individual cases is 
the natural change in circumstances between making estimates when devis- 
ing a project, and the realisation of the project. For example, the UK Home 
Office undertook a major project moving to a new headquarters (described 
in Klakegg et al. 2009). While the building process was generally a success, 
the subsequent parliamentary enquiry concluded that “There is evidence 
of optimism bias in PFI projects for departmental accommodation .... The 
Home Office assumed that staff numbers would be reduced due to outsourc- 
ing, efficiency gains, and changes to working practices. Instead, numbers 
increased dramatically” but then adds “numbers increased dramatically ... as 
the Home Office took on new responsibilities, although the total increase is 
not fully explained by these new functions”. So it is often difficult to compare 
planned benefits with the actual outcome. 


2.2.6 Stakeholders 


A practical problem is the number and range of stakeholders in public pro- 
jects, who should be consulted to identify the diverse project outcomes and 
benefits — some of which might be unknown to the government department 
at the outset. The PMI study (Williams et al. 2020a) showed that stakeholder 
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engagement and discourse were increasingly used in benefits identification, 
drawing attention to methods such as ‘benefits workshops’ to capture some 
of this discourse. This was generally seen as vital for ensuring buy-in for pro- 
jects, but some warnings were raised: questions about whether this was seen 
as ‘public relations’ — perhaps a way of legitimising a project — or whether the 
results were acted on; concerns about delaying projects; conflicts between 
stakeholders, particularly where there are different ‘tribes’ who might not 
understand each other. 

The public communication that contributed to Figure 2.1 in the A303 
project above was widespread, both for communicating the project benefits 
and for gathering stakeholder input. It was clear that stakeholders benefited in 
a wide variety of ways, since the project created a large spectrum of oppor- 
tunity for both human and non-human actors. It is to be hoped that this will 
continue, as the perception of wider and societal benefits changes over time, 
as cultural attitudes change, along with technology changes. 

Another example is given in the vignette “Ensuring the train arrives on 
time! Resolving some of the uncertainty” in Eden et al. (2005). This was 
an airport passenger transport system, a driverless train (innovative then) 
planned to move passengers both between terminals and the city at a ma- 
jor airport. Stakeholder analysis for this project, which was about to start, 
showed important aspects to consider included (for example) the views of the 
immediately local community, who had already experienced considerable 
construction disruption (and were unlikely to benefit significantly from the 
longer-term use of the airport); safety of local drivers, as the permanent way 
was built on stilts above roads that continued to operate; the views of local 
politicians and their relationship to the authority which owned the airport; 
the views of users who would transit into the city, and so on — aspects which 
should have been uncovered during the strategic development of the project. 

The nature and involvement of stakeholders will be explored further in 
Chapters 3—6. 


2.2.7 Contractors 


It is worth noting briefly that public projects are generally executed using the 
private sector. This can be simply by defining a project and then passing it 
over to the public sector to carry out. In this case, the private sector company 
acts according to the expectations laid out in the project contract with the 
government department. This will be touched on again in Section 2.4, but it 
is worth noting at this point that striving to achieve project outputs might not 
be — indeed, probably will not be — the same as striving to achieve the strate- 
gic success objectives of government departments. It is here that the delivery 
mechanism becomes important, to align the motivations of the contractor 
with the public sector partner. This is particularly relevant when, as discussed 
above, the success criteria of the public sector changes. When the Channel 
Tunnel shuttle wagons were being built (Eden et al. 2005), a major fire in 
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London (as well as the sinking of a ferry) meant that the priorities of the gov- 
ernment focused much more on fire safety, and the legislation was changed. 
This occurred in the middle of the project, meaning that the contractor — at 
that point aiming for the project outputs as defined in the original contract — 
had to make major changes to the product. 


2.2.8 Politics 


A major effect limiting clarity on ‘project success’ and causing benefit defini- 
tions to be variable is the political nature of the environment which produces 
the projects. 

There are many, many examples that could be discussed — perhaps in one 
sense any public sector project. There are many projects in many countries 
that have started as (sometimes vanity) projects for individual politicians, or 
announced by a politician unexpectedly leaving his/her Department sud- 
denly to initiate a new project. On the other hand, many other projects 
which are seemingly part of normal government business can be motivated 
or changed by political effects. One example might be the UK C-NOMIS 
system, an ambitious project planned to be a single offender management IT 
system across the prison and probation service. This is described in Klakegg 
et al. (2010, pp. 118-125), looking back to the project initiation and stating, 


this pressure on the prison system may have led to a ‘political’ agenda and 
thus political pressure to implement some kind of a solution, and then 
later on overlook warning signs. Furthermore, in this sort of environ- 
ment, often individual characters can become important in starting the 
project off. 


We will return to this example below. 

As the ‘Paradoxes’ paper drily puts it, “While the analytical process is 
largely within the realm of the professional constituency ... the decision still 
remains with the political level. And the processes and decisions at this level 
are not always rational” (p. 303). This can clearly be seen when the pro- 
ject is a significant investment (and particularly if it is high-visibility and 
high-reputation): Cicmil and Braddon (2012) refer to such projects as ‘glory’ 
projects: “... surrounded by an aura of glory through the rhetoric used to 
describe them — a narrated promise of extreme prosperity. They are often 
born out of vanity of human ambition ...” (p. 221). They analyse one particu- 
larly large (and largely unsuccessful) IT project in the UK National Health 
Service, whose size and particularly advanced technological nature gave it a 
‘glory’ aura, concluding (among many useful conclusions) that “On reflec- 
tion, the project was approved without a rational reason or, perhaps, with 
seemingly irrational reasons”. 

Politics means that the view of project success criteria can change as public 
perceptions, or ministers, change. Perhaps more notably, the timescale over 
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which projects are viewed can be quite different for a minister, looking to 
public opinion and perhaps the budgetary cycle, or even the next election, 
or a government department which might be looking at the very long-term. 
Samset and Volden’s Paradox 10, ‘the paradox of myopic decisions’, describes 
how short-term planning horizons are thus naturally brought to bear upon 
projects whose lifetime is likely to be decades. Processes within government 
departments, which have a long lifetime, should be designed to take the long 
view — but politicians who might have a short-term view have power over 
these decisions. 


2.3 Developing tactical success criteria 


Once the fundamental purpose of a project has been decided — what it is 
setting out to achieve — and quantified, the more immediate parameters 
of the project need to be settled. That is, we now need to consider the 
traditional ‘iron triangle’ tactical success criteria — timescale, project out- 
puts, and crucially, in the public world, budget. This section will introduce 
the subject, which will be explored in its different aspects in more detail in 
Chapters 3—6. 


2.3.1 Methods 


Unlike the process of defining project benefits discussed in Section 2.2, there 
is a longer history of developing processes for outlining well-defined quan- 
tified project proposals when seeking approval from government funders. 
These need at the very minimum to define the quantified project outputs, 
the way these will be achieved, the expected cost and timescale, and risks. 
Expected cost is, of course, essential in forming the basis of any cost-benefit 
conclusions. 

A good example of what is needed when developing a business proposal is 
the UK’s mandated model, the UK Treasury ‘5-case model’, which is defined 
in the Green Book and supporting guides (HM Treasury 2018a, 2018b). This 
defines five dimensions of the case that needs to be made for the programme 
or self-standing project, starting with the view from the permanent organ- 
isation (the government department), gradually getting into the temporary 
project, then at the end stepping back to the permanent organisation level — 
see Table 2.2. 

Clearly, this information is not available at the very start of project devel- 
opment, and Chapters 3 and 4 will look in detail at the logic and underlying 
theory of this process. There are generally now well-defined and mandated 
procedures in different countries to try to formalise the process. Klakegg 
et al. (2016) give some history of the project governance process in the UK, 
Norway and the Netherlands. Two examples show the current formal gradual 
refinement of the project idea. 
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Table 2.2 The UK five-case model (from the Green Book) 


Strategic case Defining why this development or change is See Section 2.2 
needed 
Economic case Choosing the best option of how to proceed See Chapter 3 


and its potential Value for Money 

Commercial case The potential commercial arrangement 
to make the proposed project happen: 
procurement strategy, defined outputs, risk 
allocation and contractual issues 

Financial case Within the proposed project/programme, 
affordability and funding 

Management case Linking back to the permanent organisation: See Chapter 7 
arrangements for delivery and monitoring; 
this should also include post-project 
evaluation 


a The UK has a three-stage process (e.g. HM Treasury 2018b), covering: 

e the Strategic Outline Case, justifying the project, filling in part of the 
Economic Case and a start of the last three cases; 

e the Outline Business Case, which identifies the best project option 
and fills in most of the Cases, ready to move on to procurement; 

e the Full Business Case following commercial negotiations ready for 
formal signing of a contract. 

b Norway has a two-stage process known as the ‘QA’ process, briefly 
described in Samset and Volden’s ‘Paradoxes’ paper. Its two steps are: 
QAI, an externally reviewed project outline required before the Cabinet 
approves the pro-project process, then a fully worked-up externally re- 
viewed proposal at QA2 required for parliamentary approval. 


This is the formally mandated process. We will explore how this process 
works out in practice in the following four chapters, but we can note four 
issues that are already clear. 


2.3.2 Estimation 


We have already noted the tendency for humans to be over-optimistic in 
their estimates both of the benefits of a project and in the cost/time, as dis- 
cussed extensively by Flyvbjerg. For him, “The root cause of cost overrun is 
human bias, psychological and political” (Flyvbjerg et al. 2018, p. 183). His 
conclusions are therefore that “Cost overrun is best avoided by (a) Getting the 
front-end of capital investments right, including using reference class fore- 
casting or similar methods to establish reliable, de-biased estimates of cost 
that fit the client’s risk appetite...” as well as (perhaps more unarguable) “(b) 
Establishing an incentive structure ... and (c) Hiring a delivery team with a 
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proven track record ...” (p. 186). Much of this paper is one step in a heated 
exchange of papers with authors led by Love (specifically, Love & Ahiaga- 
Dagbui 2018), who strongly question the basis of Flyvbjerg’s conclusions. 
One key issue here is where the ‘original budget’ is specified, as this is needed 
to consider whether there has been ‘cost overrun’: the long gestation period of 
public projects means that estimates can rise (or fall) during this period — we 
will look further at this in a few paragraphs. Another key issue is the role of 
the ‘Hiding Hand’ in projects, an idea, due to Hirschman, discussed at length 
in Ika et al. (2021) (with Love again and Pinto) pointing to “projects such 
as the Danish Great Bell Toll Bridge, the German Karlsruhe-Bretten Light 
Rail Line, the Sydney Opera House in Australia and the US Hoosac Tunnel, 
which all experienced significant cost overruns and yet exceeded benefit 
expectations”. For them, “the Hiding Hand assumes we should not presume 
we already know what success is and how to measure it”. 

A pragmatic view notes the existence of both optimism bias, ‘Strategic 
Misrepresentation’ and the “Hiding Hand’, with candid and transparent con- 
versations to ensure that these are looked out for and taken into account. As 
noted above, in Williams et al.’s (2020a) study of a number of countries, apart 
from the UK with its formal use of Reference class forecasting, there was 
recognition across countries of the issues, and clear attempts to take them into 
account, plus sensitivity analysis on cost and benefits. 

Estimation is crucially dependent upon good data — but as Chapter 5’s 
‘Conundrum 2’ states, rarely does this exist upfront in a project. However, 
Samset would regard it as often a benefit rather than a problem, and Samset 
and Volden’s Paradox 3, ‘the paradox of early information overflow’, shows 
how the over-abundance of information can be detrimental rather than help- 
ful to making a mature project estimate (this is expanded upon in Williams & 
Samset 2010). 

While estimation of time and cost is complex, a further complication in 
projects is the need to recognise the trade-off between these. Projects that 
need to be carried out quickly generally incur higher costs — and much more 
so if a project is accelerated mid-project. While this has been known for some 
time in the project world (e.g. Eden et al. 2005 and their ‘amoebic’ growth of 
project costs), it is becoming increasingly recognised in major public projects. 
Looking back on a number of projects, but specifically the UK’s roll-out of 
Broadband, the National Audit Office (2020a) reported that “attempting to 
adhere to a fixed timeline, which later proves unachievable, can contribute to 
delays and cost overruns” (p. 39), pointing to similar effects in the UK project 
to roll out smart meters and the huge Crossrail infrastructure project. 


2.3.3 Uncertainties and the nature of budgets 


As Chapter 5 notes in “Conundrum 3’, these large public projects are a com- 
plex undertaking, but budgets, particularly in public discourse, are presented 
as single, deterministic values. There are a number of problems with this. 
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First, making an estimate is a probabilistic exercise. The (epistemic) un- 
certainty is greatest at the start of the project front-end, when least is known 
about what is required and how the problem might be solved. The project 
definition is a gradual process of reducing the uncertainty in the estimate 
(see Figure 3 of the ‘Paradoxes’ paper) — but there still remains considerable 
uncertainty on the estimate even as the project starts. The Norwegian QA 
system referred to above requires explicit uncertainty statements to be for- 
mulated: at the QA2 point, 


budgets are based on formal uncertainty analyses and stochastic cost 
estimation. The recommended budget will commonly be close to the 
P85 level, and the recommended target cost for the responsible agency is 
normally lower and close to the P50 level. 

(Volden & Samset 2017, p. 97) 


However, in most regimes, the public statement of budgets does appear to be 
generally deterministic. 

Project budgets include contingency funds to cover uncertainties and risks, 
but the calculation and allocation of these funds has in the past been specific 
to any one project. This means that comparing cost overruns between pro- 
jects is difficult. To take perhaps the most well-known example: the Apollo 
moon-shot programme, which “came in at $21 billion, only $1 billion over 
its initial estimate. Few know that the initial estimate included $8 billion 
of contingencies, a thing rare in itself. Very few public projects have even 
semiformal contingency budgets ...” (Morris & Hough, writing in 1987). 
While contingencies are more formally calculated nowadays, there are dif- 
ferent treatments in different systems. In the UK, for example, “Contingency 
provision ... should be used to inform the approving authority of its potential 
liabilities. Government is self-insured and contingency should not be cred- 
ited to the approved proposal” (HM Treasury 2020). 

As well as being more informed as the process of estimation proceeds through- 
out the project front-end, the purpose of the cost estimates subtly changes. To 
put it crudely, the purpose of the very first estimate is to get approval for the 
project development process to be initiated. Once politicians have committed 
to a project, as Chapter 6 points out, it is sometimes difficult for them to change 
their mind without the risk of appearing inconsistent. The purpose of the final 
pre-project estimate is to get approval for the project to go ahead, but by be- 
coming the project budget, it is also a target by which the project will be judged 
at the end — hence Paradox 6, ‘the cost estimation paradox’, which shows a focus 
on the final estimate while forgetting about the early cost estimates. 

In a public project, where it is sometimes difficult to draw a boundary 
around the project to define what is ‘in’ and ‘out’, some growth in project 
estimates can be due to elements being included that it was not clear should 
be included at the start. A UK example was the 2012 London Olympics: 
bid at £2.4 billion in 2005 (apparently including considerable contingency, 
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Table 2.3 Cost of Scottish Parliament building project 


Up to £40 million Early thoughts 


July 1998 £50-£55 million Design chosen although site unclear. 
Figure excludes, e.g. VAT and site 
acquisition costs 


June 1999 £109 million Estimate at start of construction. 
Includes, e.g. fees, site costs, VAT, risk/ 
contingencies 

November 2001 £241 million Official announcement taking into 


account increases in space and major 
design changes and fast working 
February 2007 {414.4 million Final cost announcement 


perhaps aiming to win both the bid and public acceptance); a parliamentary 
announcement in 2006 set the budget at £3.3 billion, which by the follow- 
ing year had risen to £5.3 billion, including regeneration and infrastructure; 
then later the final budget, including contingency, security and tax, was set 
up to £9.3 billion. A final spent of £8.8 billion allowed a BBC headline to 
proclaim that the London 2012 Olympics was £500 million under budget(!). 

Looking back at early cost estimates, their increase during the project 
front-end can be shocking. The ‘Paradoxes’ paper (Samset & Volden 2016, p. 
306) gave an analysis of 12 Norwegian projects, where the first cost estimate 
of the project was compared to the final budget approved by parliament be- 
fore the start of the project: the best of these showed an increase of 70%, while 
the worst increased by 14 times, and the average increase was 650%. 

A well-known example in the UK is the Scottish Parliament building, a 
highly political project to bring a parliament to a devolved Scotland. The cost 
is summarised in Table 2.3. There are a number of different effects at play 
here: perhaps a deliberate playing down of the costs at the start to gain public 
acceptance for this political project; considerable uncertainty about what the 
project entailed — even its location; a perhaps more reasonable estimate at the 
start of the project, which included all relevant costs; considerable changes to 
the scope, which increased the project; and an undoubted lack of governance 
which enabled changes to get out of hand and for costs to overrun (a BBC 
report of evidence given to the official enquiry said that “The design of the 
[Scottish Parliament] building has been changed 15,000 times since the pro- 
ject began”). Some of this history is given in the report of the official 2004 
Holyrood Enquiry (Fraser 2004). 

It is important to disentangle these different types of effect as we look at 
project budget growth and overspend. 


2.3.4 Strategic and tactical success 


We have looked at strategic and tactical success criteria separately. In a logical 
process, the strategic criteria will be considered first, as the ‘project’ gradually 
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takes shape (although in practice, a logical process is not always followed, and 
sometimes a solution is announced, and then a logic formulated around that 
solution — see Chapter 4). But even in a properly run project, front-end, stra- 
tegic and tactical success criteria should be honed together as the front-end 
proceeds. Of course, the two are generally intimately related, and it is a trite 
observation that generally more output can be achieved with more time and 
a higher budget. 

Ifa long-term rational view is taken of a project, it could be argued that the 
strategic achievements of the project are the more important aspect, particu- 
larly as public projects tend to have long timescales, sometimes many decades. 
However, in the public arena, in a democracy, a short-term — or even an 
immediate — timescale becomes more important. Hence Samset and Volden’s 
Paradox 10, ‘the paradox of myopic decisions’: while the long term of the 
project is the more important, evaluations of projects happen, with opinions 
formed in a shorter timescale. The shortest term for an ex post project eval- 
uation is immediately on project completion, where often the benefits of the 
project cannot be seen, indeed might not have yet been realised, whereas 
the cost and timescale are immediately visible. Also, remember our point in 
Section 2.1, that public opinion concentrates more on short-term efficiency 
metrics when looking at ‘failure’ (as compared to longer-term effectiveness 
metrics when looking at ‘success’). In a slightly different context, a UK civil 
servant said, “You have to have a long-term strategy but unless it delivers 
short-term results no one will believe you” (Sir Michael Barber, head of the 
Prime Minister’s Delivery Unit) (Barber 2007, p. 75). This effect will follow 
us as we move on through the project. 


2.4 During the project 


While this book is concentrating on planning and developing major public 
projects, it is important to consider how these ideas of ‘success’ permeate a 
project as it moves into execution — and indeed post-project evaluation — 
since good efforts at the start of the project might be nugatory if the exe- 
cution of the project pulls in a different direction. We have to plan in good 
practice at the start of the project. 


2.4.1 Concentration on the tactical 


Much study has shown one effect very clearly. During the start-up and de- 
velopment of a well-formed public project, there should be a lot of attention 
on the strategic aims of the project: what need it is fulfilling, why it is good 
for the country to try to gain the project outcomes. As discussed in the pre- 
vious sections, increasingly formalised project approval procedures within 
countries have developed processes that require project sponsors to justify 
the project in terms that are in line with the strategic aims of the govern- 
ment or department. Finance departments or treasuries will not agree to 
projects being funded unless they are justified in terms of the (financial or 
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non-financial) gains that will be achieved. Projects should now be started 
with a clear vision of what the project is setting out to do and why. 

However, the emphasis on the strategic aims of the project often dissipates 
once funding has been granted and eyes external to the department have been 
taken off the project. À multi-country study by Williams et al. (2020a) dis- 
cusses the emphasis on benefits identification “as a means of getting the pro- 
ject through the approvals process” and continues, “Consideration of benefits 
tended to fade once funding was achieved ... there seemed to be a skew to- 
ward project delivery (particularly project-management success) rather than 
benefits after project sanction, excepting occasionally there was an increased 
focus on benefits at project closure as benefits were evaluated and reported” — 
although they do note exceptions (see below). As a UK parliamentary report 
quotes more informally, 


We also have the impression now where the emphasis ... is on the de- 
livery of the project as defined: getting it on time, on cost, as defined at 
the beginning, and the actual benefits that the project is there to deliver 
sometimes get — I will not say lost but there is less priority put on that 
than the actual delivery of the project. 

(House of Commons PACAC 2019) 


Governance processes should try to minimise this effect, for example using 
formal reviews of projects. However, a detailed review of a major database of 
reviews of the biggest public projects in the UK, described in Vo et al. (2021), 
showed that of all the recommendations made, 70% addressed delivery issues 
and only 30% concentrated on the higher-level effectiveness success criteria. 
In keeping with Williams et al.’s findings, 


PVRs [independent reviews supporting project initiation] and project 
closure reviews had a slightly higher percentage of recommendations 
focusing on benefits, perhaps implying a skewed emphasis on benefits 
towards project initiation (to get projects started) and closure (to get pro- 
jects signed off). 


Vo et al. (2021) continued, “recommendations were much more linked to 
what was directly needed to get to the next stage of the project cycle, rather 
than project benefits”. Chapter 4 will re-visit this tendency of assurance to 
focus on the process of completion against arbitrary budgets rather than the 
strategic aims of the project. Of course, we are again echoing Samset and 
Volden’s first Paradox: the danger of measuring success in terms of tactical 
performance rather than achievement of the strategic aims of the project. 
These dangers are there, even if the strategic benefits have been assessed 
thoroughly at the start of the project. If, however, the initial project logic 
itself is weak, it leaves the project in even more danger of not delivering 
a useful output. Returning to the UK C-NOMIS prison/probation IT 
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system described above (Klakegg et al. 2010), this was a project with ambi- 
tious strategic objectives to “Improve positive offender outcomes (1.e. reduce 
re-offending) ... introduce more assertive case management ... Integrate IT 
support ... [and] improve means of monitoring compliance”. Clearly, a major 
project, developing a major business change to end-to-end supervision of 
the individual offender. However, as reported by the National Audit Office 
(2009), the team “treated C-NOMIS as an IT project rather than a major 
IT-enabled business change programme” — which meant that they “did not 
get to grips with the business changes required to design and implement a 
single offender database across both services”. Furthermore, even within the 
narrow confines of an IT project, as a good example of Samset and Volden’s 
Paradox 4, ‘opportunity space’, 


there were .... Other possible solutions not explored fully .... There is no 
evidence of the team considering factors such as the nature of support- 
ing infrastructure and the existence of common levels of service, which 
should have informed the selection of the technical solution. 


Initially, the project had an approved lifetime cost of £234 million to 2020. 
By six months before the original planned completion date, £155 million had 
been spent, the project was two years late, and estimated lifetime project costs 
had risen to £690 million. The project was then halted. 

To give a balanced view, we should remember the issues discussed towards 
the end of Section 2.1: public projects live in a world of turbulence, and if 
they reacted to every change in government viewpoint or public opinion or 
whim, then management of the project would be impossible. The discipline 
of project management is there to try to bring order within the chaos, but a 
project impervious to the strategic aims of its owner risks losing its way and 
becoming one of those projects successful in ‘efficiency’ terms, but useless, 
like the on-shore torpedo battery or Zwentendorf Nuclear Power Plant, dis- 
cussed at the start of the chapter. 


2.4.2 Organisation and roles within the project 


Within the project, there is an increasing emphasis on the use of methods 
to maintain attention on the strategic aims of the project, generally coming 
under the heading “Benefits Management’. Williams et al. (2020a) sets out 
these ideas in various countries and shows the increasing interest as attention 
shifts from ‘project management’ to the strategic aims of projects. This study 
also identifies common barriers to this approach, such as lack of senior man- 
agement buy-in, lack of a benefits culture and the lack of any requirement for 
benefits oriented ex post analysis, as well some enablers to the approach such 
as increasing stakeholder engagement and clarity in accountability. 

Of course, many national project structures now generally include some 
system of in-project reviews, designed to bring an ‘outside view’ to the 
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project. In the original UK system, there is a clear system of ‘Gateways 1-5? 
during the project; however, this is overlaid by the ability to carry out a 
‘Gateway 0’ at points during the project, designed to take a step back and 
consider the continuing relevance of the business need and alignment of the 
project with that need (see the comparison in Klakegg et al. (2016) of the UK, 
Norway and Netherlands systems). Interestingly, the ‘exceptions’ to the phe- 
nomenon of fading interest in strategic success during a project noted earlier 
included, among others, “projects that had to go through the NSW’s ICT 
gateway process or the UK IPA assurance process ...” (Williams et al. 2020a). 

At the highest level, in many jurisdictions, projects are undertaken by a 
separate body to the government department. In the UK, 


Major government infrastructure projects in the UK are most commonly 
started, approved, funded and overseen by a sponsoring Department of 
State .... They are normally delivered through arms-length bodies (ALBs) 
of a range of forms .... It is the delivery organisation’s job to take the 
requirements of the sponsor, turn them into specifications, contract for 
their delivery and secure the intended outcomes to time, quality and cost 
through their private sector supply chain. 

(Department of Transport and IPA 2019) 


Further, 


This separation of functions allows Departments to specialise in gov- 
ernment policy and legislation whilst the delivery organisation focuses 
on project delivery through its contracted supply chain and advisors. 
This division has significant advantages but can also create boundary 
issues and sometimes cultural challenges between the organisations. .... 
Different sorts of issues can arise through the project lifecycle. 


The report goes on to identify 24 lessons drawn from a number of case studies 
on how to sponsor such projects. 

One straightforward method which seems to have worked well within 
one of these arms-length bodies, Highways England, is for a project to have 
two directors reporting to the officer accountable for the project: a project 
director responsible for the delivery of the project in traditional terms and a 
sponsorship director (Highways England 2018) responsible for realising the 
benefits of the project. The creative tension between these two appears to 
lead to a concentration on both aspects of success. Our statement of this spon- 
sorship director role actually downplays it. For Highways England, sponsor- 
ship directors, 


act as a conscience and guide to delivery teams. While operating outside 
direct day-to-day delivery activities, they provide strategic oversight and 
retain accountability for the business case and outcomes, whilst ensuring 
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assurance throughout the project lifecycle .... Above all, a sponsor must 
always maintain an unremitting independent focus on the true reasons 
and benefits for which the project is being undertaken and how these can 
be achieved. 

(Highways England 2018, p. 6) 


The effective analysis of the A303 project above is an example coming from 
this organisational structure. 


2.4.3 Other actors 


A project does not sit in a vacuum, looked after by the government depart- 
ment, with no influence from the outside. 

First, a project is generally prosecuted through a private sector partner. 
As mentioned in Section 2.2, in terms of tactical success, the contractor will 
be aiming for the success criteria laid out in the contract for the work. At a 
strategic level, the company will not have the same aims as the government 
department. On the other hand, a company will not be subject in the same 
way as the government department or arms-length body to the vagaries of 
public opinion or politics. 

The difference in lack of strategic alignment is illustrated by two exam- 
ples. The first is the long and sorry story of the NHS IT project analysed by 
Cicmil and Braddon (2012) (see Section 2.2), concentrating on the “small 
number of key suppliers, each of whom had a different business agenda to be 
pursued and objectives to be gained from their involvement in the project”. 
The second is the Acela programme in the US: Amtrak was going through 
fundamental financial issues with questions about their strategic direction 
during the Acela programme, causing huge disruption to the programme, 
and eventually the train manufacturer sued Amtrak, 


seeking to recover $200 million in damages .... Designs have been mod- 
ified literally thousands of times .... Amtrak scheduled multiple public 
relations visits to a test track; those visits disrupted operations in a quest 
to hype Amtrak’s bright future and minimize public recognition of defi- 
ciencies in train design and program administration. 

(Vranich et al. 2002) 


Understanding these differences in strategic aims is even more important in 
larger projects where consortia of companies are involved, or where compa- 
nies interact, since projects do not exist in a vacuum. Gil and Pinto’s (2018) 
analysis of four major UK projects (HS2, London Crossrail, London 2012 
Olympics and Heathrow T2) talks about the projects being set within “Lon- 
don’s megaproject ecology”. 

Second, the public will often have a crucial role, particularly for a public 
project ultimately reporting to politicians. Their involvement will depend 
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upon the type of project. Any infrastructure project will involve local and 
other interested people: the A303 project described above consulted with a 
wide range of interests. There is a general move within democracies to con- 
sult and involve the public. Gil and Pinto (2018) continue by saying that large 
infrastructure projects, in contrast to more technologically complex settings 
such as aeronautical projects, 


are socially complex but not so technologically complex that planning 
choices cannot be comprehended by multiple heterogeneous stakehold- 
ers. The fact that many actors can grasp what the issues are and what is 
at stake exacerbates the interdependency with the environment. Hence, 
a choice to set up a polycentric system responds to growing calls in the 
environment for organizations to adopt more collaborative and inclusive 
decision-making processes. 


Public opinion can be fickle, and the mood on particular investments 1s af- 
fected by media reporting. More fundamentally, public attitudes to criteria 
can change over time, such as to the environment, climate change, air pol- 
lution, crime, social cohesion or (remembering the A303 example) heritage: 
even a good decision-making process at the start of a project can become out 
of date if the public opinion weightings of these criteria change over time. In 
addition, as Chapter 5’s ‘Conundrum number 4 will discuss, during a project 
the public sometimes focus on efficiency measures, particularly cost, rather 
than the benefits of the project, so the pressure we have discussed to meet 
efficiency targets is to some extent driven by the public through parliamen- 
tary processes. 

This is overlaid by the temporal cycles of government: four-yearly elec- 
tions, annual budgets and regular spending reviews. These are asynchronous 
with the sometimes long project lifecycle, adding to the turbulence around 
the project as strategic and spending priorities frequently change. 

All of these effects mean that an important role for project sponsors is to 
keep the public ‘on side’ during a project, particularly as the environment 
changes, and there is a constant need to re-translate the project in terms the 
public will understand — assuming, of course, that it is still relevant to the 
needs of the country, as discussed above. 


2.4.4 A final note: accountability 


We have discussed at length the various parties who take some degree of 
responsibility for a project — but who at the end is accountable for delivering 
the outcome that caused the project to be set up in the first place? In the UK 
system, it is the ‘senior responsible owner’, a clearly defined role (Infrastruc- 
ture and Projects Authority 2019) who “is accountable for ensuring a pro- 
gramme or project meets its objectives, delivers the projected outcomes and 
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realises the required benefits”, and for the most major projects (i.e. those in 
the Government Major Projects Portfolio), 


as well as being accountable to their own organisation’s management, 
also has personal accountability to Parliament for the implementation of 
the government’s policies as assigned to them by the relevant accounting 
officer. This accountability is recorded in the senior responsible owner’s 
letter of appointment. 


This seems very clear. However, while a project manager’s efficiency targets 
are generally explicit and unambiguous, we have seen over this chapter that 
there are a number of issues that make the achievement of strategic objectives 
much less clear and more contested. As we look in the next section about 
reviewing the project ex post, we shall see how difficult it often is to be able 
to say definitively “this project did (or did not) achieve its aims”. 


2.5 Post project 


Just as we turn at the end of a project to look back to the original defini- 
tions of success, to judge how well we have done, so we must look back at 
our original definitions of success in the earlier sections to consider what a 
post-project evaluation of success means. 

Having said that, the evidence suggests that there is a lack of ex post eval- 
uation in practice. The review of Williams et al. (2020a) suggests less activity 
than might have been hoped for, for various reasons. ‘Lessons learned’ or 
post-project reviews looking at the project management and efficiency meas- 
ures are becoming at least not unusual, but this is not the same as an evalu- 
ation which considers the effectiveness or benefits of the project. However, 
Williams et al. (2020a) do note some exceptions: National Audit Offices cer- 
tainly look at the value of projects, and public scrutiny of public expenditure 
is perhaps increasing the appetite for such reviews. 

That said, as we commented at the end of the last section, there are a 
number of issues that need to be taken into account when taking a view on 
a delivered project. 

First and most simply, public projects often have long lifespans, and the 
world — particularly the political world — will change during the lifetime 
of the project. This means that the value of the project objectives laid out 
at the start of the project might have changed. There might be governance 
questions — why did we not halt or change this project mid-stream? — but it 
will not be unusual for a simple comparison of planned outcomes and actual 
outcomes to founder on the passage of time. 

A second point particularly applies to projects in IT, transformation or 
the military. At the point that an IT or transformation project finishes, the 
project output (say, an IT system) generally has reaped no benefits at all: it 
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passes to the department and gets used, entering ‘business as usual’. For our 
purpose of looking at project ‘success’, this raises two issues. First, if a senior 
responsible owner has passed an output over to the sponsoring department, 
how can he/she be responsible for the use that is made of the system? Indeed, 
second, if it is just part of ‘business as usual’, where is the need to monitor it? 
A review by the National Audit Office (2018) of projects which had left the 
Government Major Projects Portfolio said, 


There is a varied picture as to whether projects have delivered success- 
fully after they leave the Portfolio. Once projects leave the Portfolio, the 
Authority is no longer responsible for monitoring progress in delivering 
benefits, it is up to sponsoring Departments to provide this oversight. 


Indeed, for four projects, 


it was unclear what had been delivered because Departments had stopped 
monitoring them, due to either a change of policy or because the De- 
partment had decided to deliver them in different ways, which resulted 
in project teams being disbanded and so Departments were unable to 
answer ... questions. 


Whyte and Nussbaum (2020) looking at mega-projects such as Heathrow 
Terminal 5, the London 2012 Olympics, and London’s Crossrail, focus on 
this boundary between ‘the project’ and ‘operations’, and see an array of 
problems which can occur, proposing for example “strategies for mobilizing 
artifacts, procedures, soft landings, and tests” (p. 506). 

Moving beyond this question of project handover, our third point is that 
sometimes it is not even fully in the Department’s hands to realise the benefits 
of a project. 

Some projects, such as new public-facing IT systems, will depend upon 
public take-up. This chapter, for example, is being written in the UK dur- 
ing the Covid-19 pandemic. The government decided to offer its citizens a 
‘phone app’ for contact tracing, local area alerts and venue check-ins. A first 
version was abandoned in May 2020 due to technical failings; the trial of a 
second app (based on Apple and Google’s technology) started in August 2020 
and was launched across England and Wales on 24 September. There was 
widespread scepticism about whether it would be used, but the government 
was able to announce by late December 2020 that the app had been down- 
loaded 20.9 million times (UK Government 2020) — an apparent success 
(although it does not necessarily show whether citizens actually use the app). 

Some projects facilitate access to benefits, but these will not be realised 
unless other bodies take them up. Transport infrastructure is a clear example 
of this. An ambitious plan was devised in the UK to develop infrastructure 
to link the cities of Oxford and Cambridge. While essentially an infrastruc- 
ture development plan, a report by the National Infrastructure Commission 
(2017) describes essential development and governance proposals to facilitate 


Project success 37 


bodies getting together from Oxford and Cambridge and towns in between, 
such as Milton Keynes, to build on the development; however, any nationally 
commissioned infrastructure work would not achieve its planned benefits if 
the local authorities did not take up the opportunities provided. 

Benefits can be even further from the immediate purview of the ‘project’. 
In the example in Chapter 5, the UK Ministry of Justice has as its main stra- 
tegic aim ‘A prison and probation system that reforms offenders’. Part of this 
development is a ‘Prison Estate Transformation Programme’. A key aim of 
the programme will therefore be to develop the prison estate in such a way 
as to reform offenders and reduce re-offending. The logical inference would 
be that when re-building an old prison, a strategic project target would be a 
reduction in eventual reoffending rates. But in this, as in all of the cases de- 
scribed here, the ‘success’ of the project will be dependent upon changes in 
the behaviour of citizens, business, government agencies, civil servants and 
other relevant stakeholders (see the discussion on social bonds above). 

Fourth, a key difficulty in evaluating a public project with a long life cycle 
aiming to bring economic benefits is the challenge of distinguishing benefits 
that arose from the specific project being considered from macroeconomic and 
other trends. The question of what improvements arose from the existence of 
the project, and what would have happened had the project not been under- 
taken (let alone if the project had not been undertaken, but the money spent 
elsewhere), is often impossible to answer convincingly. Some jurisdictions we 
have surveyed do not generally try to do this, because of the contested nature 
of any answer that might arise. The project business case should have tackled 
this to some extent, but this disentanglement is clearly difficult. The problem 
is made more complex by raising the issue of when post-project benefits should 
be assessed: the quicker the assessment, the easier it will be to see the imme- 
diate effects of the project, but a longer-term is needed to understand whether 
the project was worthwhile. This is reflected in Samset and Volden’s Paradox 
10: “projects that are meant to last for decades and sometimes centuries may 
have significant impact on economic, environmental, and social development, 
yet they are still assessed in a short-term and static perspective” (p. 309) — 
because the public sector wants to know whether the project was worthwhile, 
without waiting for the long-term perspective that history provides. 

So, fifth, it is historical reflection and public opinion that provides the 
long-term judgement on projects, particularly as the emergent and fluid na- 
ture of benefits diverge from the pre-defined project. Projects that are seen 
as a failure at the time in almost every way, such as the London Millennium 
Dome, can be seen later to be a success (with, as always, the focus on longer- 
term success criteria while the ‘iron triangle’ fades into history). 


2.6 Conclusion 


As we go on to explore the development of major public projects in practice, 
this chapter has tried to set the scene by considering what we mean by ‘pro- 
ject success’. 
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At the most basic level, we have distinguished between tactical project 
management efficiency success and strategic project output effectiveness (and 
longer-term) success. We have seen Samset and Volden’s paradox that, in 
practice, minds tend to be focused more on efficiency targets than the effec- 
tiveness targets that were the reason for the project — also when the public 
thinks about ‘failure’ rather than ‘success’. Particularly once a project has 
been approved and is underway, attention drifts away from the strategic aims 
to the delivery of the pre-defined output, sometimes only drifting back when 
the project is near completion. 

As we try to define a project’s strategic aims, we have seen that many are 
contested and difficult to quantify. In public projects, there is often an ar- 
ray of different stakeholders, with different objectives and different ways of 
talking about aims and objectives. Even for quantifiable outcomes, we have 
seen different reasons for drawing up budgets and the effects of human bias, 
politics and interests in drawing up estimates, which will be discussed further 
in Chapter 5. We have seen how the very idea of a project ‘benefit’ is fluid, so 
we need to take a more fundamental look at the project front-end; Chapter 4 
will re-visit this. 

We have seen how projects, designed to be self-contained with clear tar- 
gets, sit in an environment which is inescapably turbulent, subject to political 
influences and often working on different (shorter) timescales than the pro- 
ject. Chapter 6 will look further at politics and incentives. We have seen some 
of the difficulties in comparing ex-ante estimates with ex post out-turns, and 
we have noted Samset and Volden’s paradox about looking at very long-term 
projects in terms of their immediate value; this will be considered further in 
Chapter 7. 

The following chapter will now look at the logic of the front-end and de- 
scribe an effective front-end process. 
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3 The logic of the project 
front-end 


Ofer Zwikael and Alicia Gilchrist 


3.1 Introduction 


Although the front end has been shown to be critical to the strategic 
success of the project, this phase of the lifecycle is not well understood. 
(Williams et al. 2019) 


The first phase of a project is known as the ‘front-end’ or initiation (e.g. 
Zwikael & Meredith 2019), followed by the planning, execution and finalisa- 
tion phases (Besner & Hobbs 2006). The front-end phase begins with a prob- 
lem or an opportunity formulated into a business case, and ends when it is 
approved or rejected by a funding entity (Edkins et al. 2013; Kock et al. 2015; 
Verworn et al. 2008; Williams & Samset 2010). The purpose of the front-end 
phase is to select the best project idea, define the objectives of the project, 
assess its feasibility, and set the project up for tactical and strategic success 
(Williams et al. 2019). Decisions made during the front-end impact the pro- 
ject’s outcomes and key stakeholders (Morris 2013; Pinto & Winch 2016). An 
effective front-end can help mitigate risk, improve cost and schedule perfor- 
mance, and enhance project value generation (Chenger & Woiceshyn 2021; 
Edkins et al. 2013; George et al. 2008; Kolltveit et al. 2004; Williams et al. 
2019). Conversely, an ineffective front-end may result in poorly defined pro- 
ject requirements, an ineffective project prioritisation and selection process, 
and, ultimately, approval of a project that is sub-optimal and will not deliver 
the anticipated benefits. 

However, there is little consensus within the literature as to which activ- 
ities should be accomplished during the front-end, and how the activities 
should be ordered (Edkins et al. 2013). This lack of clarity may be a result 
of confusion as to where the boundaries of the front-end lie — is it a phase of 
the project or does it occur before a project formally commences? (Williams 
et al. 2019); confusion as to who is accountable for the front-end; the inher- 
ently ‘fuzzy’ and developmental nature of the front-end (Kim & Wilemon 
2002; Takahashi et al. 2018); variability in front-end activities between indus- 
tries and projects (Chenger & Woiceshyn 2021); and the high risk of lock-in, 
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whereby decision-makers unofficially commit to a project before a formal 
decision is made (Cantarelli et al. 2012). This lack of clarity may, in part, be 
a reflection of the project management literature and bodies of knowledge, 
which place heavy emphasis on the planning and execution phases of a pro- 
ject but tend to overlook the front-end phase (Edkins et al. 2013). Following 
the lack of consensus in the literature, and the growing importance of man- 
agement of the front-end phase in research and practice, our first research 
question is, How should an effective project front-end phase be constructed? 

Although what constitutes an effective front-end phase is unclear or ‘fuzzy’ 
(Chenger & Woiceshyn 2021), it is accepted in the literature that this phase 
begins with an idea (Verworn et al. 2008) and that the idea for a new project 
is typically generated by a problem or an opportunity (Baker et al. 1967; 
Samset & Volden 2016; Williams et al. 2019; Zwikael & Smyrk 2019). We 
consider a good project idea to be one that is likely to contribute towards the 
achievement of organisational outcomes by enhancing strategic performance. 
Yet how a good project idea is generated is unknown. This research gap is im- 
portant because strategic project failure can often be “traced back to decisions 
in the earliest phases, when the initial idea was conceived and developed” 
(Edkins et al. 2013; Samset & Volden 2016, p. 301). Therefore, our second 
research question is, How are good project ideas generated? 

Samset and Volden (2016) identified ten paradoxes that are found to occur 
within the front-end of a project. Of these ten paradoxes, four are of particu- 
lar relevance to this chapter — they are known as ‘the paradox of the signifi- 
cance of front-end management’, ‘the paradox of the opportunity space’, ‘the 
paradox of strategic alignment’ and ‘the paradox of “predict and provide”. 
We examine these paradoxes to highlight key problem areas within the front- 
end phase and to contribute towards their future resolution by investigating 
effective front-end management. Subsequently, we address a third research 
question: How can effective front-end practice contribute to the resolution of common 
front-end paradoxes? 

Our research addresses the three research questions and contributes to the 
knowledge by investigating advanced project practice. This chapter details 
the front-end process followed by the Australian Defence Force (Defence) 
to generate new project ideas, prioritise them and select the projects most 
likely to achieve the strategic objectives. Defence presents an ideal case study, 
as the front-end of a Defence project is rigorous, protracted and advanced — 
and continues to mature (Department of Defence 2020a). Furthermore, large 
Defence projects outperform private similar size sector projects that are less 
complex (Cook & Unewisse 2020). Using document analysis and a hypo- 
thetical case study of a Defence project, this chapter advances the project 
management literature by extending our knowledge of the front-end process 
through an in-depth investigation of what is widely considered to be an ef- 
fective front-end process. 

The remainder of the chapter is structured as follows. In the next section, 
we discuss four common front-end paradoxes. In Section 3.3 we review the 
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project front-end literature. Our methodology is then detailed in Section 3.4, 
before we present our findings in Section 3.5. Our findings and their the- 
oretical and practical implications are discussed in Section 3.6, before we 
conclude the chapter in Section 3.7. 


3.2 Project front-end paradoxes 


How the objectives of the front-end phase should be achieved to produce the 
best outcomes for the project and key stakeholders is not well understood in 
the literature. This is evidenced, in part, by the presence of the four paradoxes 
discussed in this section. It is also demonstrated by the linkages between de- 
cisions made during the front-end phase and the explanations that are most 
often provided for project failure (Caldas & Gupta 2017; Kock et al. 2016; 
Williams et al. 2019). In this section, we discuss each of the four paradoxes 
and consider the various reasons as to why each of the paradoxes may persist. 


3.2.1 Paradox of the significance of front-end management 


The critical role of the front-end phase in affecting the outcomes of a project 
is widely acknowledged and yet this phase is “unclear and poorly understood” 
(Williams et al. 2019, p. 1140). As a result, the project management litera- 
ture does not sufficiently consider the larger context in which the project is 
“idealized, validated, and shaped by multiple stakeholder forces” (Pinto & 
Winch 2016, p. 238). As Samset and Volden (2016) point out, this presents 
a paradox, known as the ‘paradox of the significance of front-end manage- 
ment’, as it is during the front-end that the opportunity to reduce project 
risk and uncertainty is greatest. This paradox can be partly explained by the 
uncertainty that characterises the front-end (Kolltveit et al. 2004), the risk 
that any investment in this stage may be lost if the project does not go ahead 
(Morgan 1987) and the lack of understanding as to what should be done dur- 
ing the front-end phase (Pinto & Winch 2016). 


3.2.2 The paradox of the opportunity space 


Once the trigger for a project has been identified, an idea can be generated to 
respond to the problem or opportunity in the form of a project. At this stage, 
the project champion should, ideally, generate a number of different ideas to 
create a range of options. Zwikael and Meredith (2018, p. 485) define the 
project champion as the person “who leads the development of the business 
case and typically presents it to the funder for approval”. The project funder 
is the “person with the authority to approve the project and commit resources 
for its execution” (Zwikael & Smyrk 2015, p. 854). Samset and Volden (2016, 
p. 302) identified a paradox that often occurs during this activity, which they 
labelled “The paradox of the opportunity space’ whereby “the choice of con- 
ceptual solution is made without systematically scrutinizing the opportunity 
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space up front”. According to this paradox, project champions often system- 
atically select particular ideas and avoid others — even when there are more 
rational choices available. 

This paradox is evidenced through a case study by Samset et al. (2014), 
which found that 11 of 17 major public projects already had a choice of con- 
cept when the front-end commenced. Samset et al. (2014, p. 474) concluded 
that the “final choice of concept is determined more by decision makers 
than by analysts, and will often be the result of policy and preferences more 
than objective reasoning”. There are a number of explanations as to why 
the paradox of the opportunity space exists, including bounded rationality, 
the tendency of discipline experts to focus on particular aspects and over- 
look others, political pre-determination, experiences and preferences of the 
decision-makers, traits of the organisation, and a high level of project specific 
detail involved in the analysis (Samset et al. 2014). 


3.2.3 The paradox of “predict and provide”? 


The paradox of the opportunity space is closely related to another paradox 
found to occur during the front-end known as ‘The paradox of “predict and 
provide” whereby the tendency is to choose a “predict-and-provide strategy 
rather than explore alternative solutions” (Samset & Volden 2016, p. 308). 
The authors illustrate this paradox with an example: when responding to a 
capacity problem, planners are more likely to select solutions that increase ca- 
pacity rather than to explore the opportunity space and consider alternatives, 
such as how to solve a congestion problem. They attribute this paradox to 
the decoupling of needs and benefits analyses from the most pressing political 
priorities and goals, and suggest this occurs due to conflict or complexity 
arising between the two. This paradox may also be attributed to path de- 
pendence (Arthur 1989; David 1985), which infers our “choices are condi- 
tioned by decisions we have made in the past, and that these decisions create 
increasingly constrained processes that cannot easily be escaped” (Aaltonen 
et al. 2017, p. 749). An alternative, contrasting perspective known as ‘path 
creation’ was proposed by Garud and Karnoe (2001) to address a limitation 
of path dependence, which is that it does not account for the role of human 
agency in forging new paths (Aaltonen et al. 2017). 


3.2.4 The paradox of strategic alignment 


To effectively evaluate and prioritise a project idea, and for the selected project 
idea to be developed in an effective project, it is necessary to clearly articulate 
the objectives of the idea, and for the objectives to be in alignment with the 
organisation’s strategy (Patanakul & Shenhar 2012). Project-strategy align- 
ment should persist throughout the lifecycle of a project. However, in prac- 
tice, strategic project goals are often not realised. In addition, while widely 
recognised as important, project strategy management which “systematically 
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relates project definition and development to corporate goals and strategies” 
(Morris & Jamieson 2005, p. 16) is not often used in the practice of project 
implementation (Patanakul & Shenhar 2012). This is supported by Young 
and Grant (2015), who found that in ‘normal environments’ only one in 
five strategies were positively impacted, while in highly collaborative envi- 
ronments where the strategic goals were stable, two in five strategies were 
positively impacted. The authors recommend future research to explore how 
projects can better contribute to strategy. 

Samset and Volden (2016, p. 304) consider this a paradox as “strategy and 
alignment of objectives are highlighted as essential concerns, but in most 
cases the internal logic of causalities and the probabilities of realization are 
erroneous” (Samset & Volden 2016, p. 304). The authors labelled this “The 
paradox of strategic alignment’. One explanation for this paradox may be 
that projects often lack shared objectives and agreement on the objectives, 
having too many goals, or too many unrealistic or overly ambitious goals 
(Samset and Volden, 2016). In addition, the changing environment makes 
project-strategy alignment, if obtained, difficult to maintain throughout the 
project’s development. 


3.3 Project front-end activities identified 
in the literature 


A comprehensive review of the literature was conducted using the Scopus 
database. The literature search was confined to journal articles published 
between 2009 and 2021 to capture the latest research. We used the key words 
‘project management’, and ‘front-end’ or ‘initiation’, and further confined 
the search to articles’ title, abstract or keywords. Each article was scanned to 
confirm its relevance to this research. This process reduced the number of 
articles from 183 to 32. We then examined each article to identify activities, 
stages or steps associated with the front-end phase of a project. Our analysis 
of the literature suggests that the logic of the front-end involves four key 
activities: (1) project trigger identification, (2) project idea generation, (3) 
business case development and (4) business case appraisal. These activities 
are presented in Table 3.1 and further discussed in this section. To provide a 
theoretical basis to the front-end project activities identified in the literature, 
we aligned these four activities with Mintzberg et al.’s (1976) three-phase 
executive decision-making process presented in the strategy literature — 
these phases appear in the last column of Table 3.1. We conclude that the 
decision-making phase of opportunity/problem identification is equivalent 
to the front-end activities of project trigger identification and idea genera- 
tion; the solution development decision-making phase is equivalent to the 
project business case development; and the selection phase is similar in nature 
to the appraisal of the project business case. Therefore, we argue that the pro- 
posed project front-end process is well aligned with the strategy management 
literature. 
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Table 3.1 Project front-end activities 


Decision-Making Project Funding Description Key Steps 
Phases (Mintzberg Decision-Making 
et al. 1976) Activities 
1. Opportunity/ 1. Project The identification + Trigger identification 
problem trigger ofa problemor + Trigger confirmation 
identification identification opportunity 
2. Project idea The generation e Idea definition 
generation of an idea to e Idea assessment 
respond to a e Idea prioritisation 


project trigger. ° Project idea 
conceptual approval 


2. Solution 3. Business case The development + Definition 
development development of a business case * Analysis 
for a new project + Packaging 
3. Selection 4. Business case Decision-making + Business case 
appraisal regarding presentation 
the proposed e Business case 
business case assessment 
e Project funding 
approval 


e Next steps decided 


3.3.1 Project trigger identification 


The first front-end project activity is project trigger identification. A new 
project is triggered by a stimulus which is most often a problem or an oppor- 
tunity (Baker et al. 1967; Williams et al. 2019; Zwikael & Smyrk 2019). How 
a project idea is triggered is important because a new project idea should be 
guided by the anticipated effect rather than the present undesired situation 
(Williams & Samset 2010). Thus, the trigger for a project idea should de- 
termine the anticipated effect. However, according to Zwikael and Smyrk 
(2019, p. 4), “although it is necessary to align projects with organizational 
strategy, not all projects that an organization funds arise directly from the 
demands of a strategic vision”. The authors argue that projects can be trig- 
gered as either (1) deliberate strategy implementation projects that implement 
an organisation’s strategy, (2) emergent strategy projects that are triggered by 
opportunities to enhance performance and shape the organisation’s strategy 
or (3) imposed projects that are demanded by external factors, but do not 
necessarily align with an organisation’s strategy. A lack of integration be- 
tween a project and organisational strategy is linked to project failure (Kock 
et al. 2016). This research supports Samset and Volden’s (2016) paradox of 
strategic alignment. 

Identifying the trigger, and understanding the effect required in response 
enables the generation and prioritisation of project ideas, an assessment of 
the alignment between the project’s goals and the organisation’s strategic 
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objectives, and an assessment of the overall value of the project to the organi- 
sation. However, how an organisation can identify project triggers most effec- 
tively and comprehensively is not explicated within the project management 
literature “contexts where exploration of the unknown and highly uncertain is 
required to identify high-risk, high reward project opportunities” (Chenger & 
Woiceshyn 2021, p. 176). Furthermore, the implications of an imposed project 
that does not align with an organisation’s strategy are also poorly understood. 


3.3.2 Project idea generation 


The second activity to occur during the front-end phase is project idea gen- 
eration. There are five reasons why the project idea generation process should 
be understood. First, strategic project failure has been linked to the concep- 
tion and development of the initial idea (Edkins et al. 2013; Samset & Volden 
2016). Second, project ideas tend to survive the ensuing prioritisation and 
selection process, regardless of how poor they may be (Samset & Volden 
2016); therefore, it is critical that the idea generated at the very start of the 
front-end process is the right one (Kock et al. 2015). Third, understanding 
the idea generation process can ensure that the trigger and the anticipated ef- 
fect are clearly defined, that the anticipated effect is triggered by the problem 
or opportunity, and that the two are not conflated. Fourth, once a process is 
known, it can be improved. 

Finally, while it is necessary to execute the front-end of a project effec- 
tively, it is also necessary to understand how a project came to be “idealised, 
validated and shaped” during the front-end so that its execution can be situ- 
ated within the project’s larger context (Pinto & Winch 2016, p. 238). This 
situational awareness can facilitate the project’s effectiveness by contextual- 
ising the project manager’s knowledge of the project specifications and cus- 
tomer needs (Morris 2013). Knowledge of how a project idea was generated 
can situate the project within its larger context by providing information as 
to how it was idealised, validated and shaped. 

There are several reasons as to why a poor project idea might survive the 
project appraisal and selection process, and receive funding. Sleesman et al. 
(2012) conducted a meta-analysis and presented a comprehensive overview of 
the determinants that provide an explanation as to why escalation of commit- 
ment to losing courses of action occurs. They found the reasons individuals 
may continue to invest in an idea, regardless of how poor it might be are 
either project-related (such as decision risk and opportunity cost information), 
psychological (such as sunk costs and ego threats), social (such as resistance to 
others’ decisions) and/or structural (such as agency problems). Cantarelli et al. 
(2012) describe the unofficial commitment to a project before a formal deci- 
sion is made as ‘lock-in’. This desire to commit may stem from several sources 
of pressure, such as fear of being ‘overtaken’ by competitors; fear of missing a 
window of opportunity; the need to respond to a threat; pressure to respond 
to senior members’ demands to find a solution to an ongoing and expensive 
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problem; the pressure to perform and ‘be seen to be doing something’; or to 
the mistaken belief that the best solution has been found, and therefore fur- 
ther testing and analysis of alternatives solutions would be a waste of resources 
and time. The project idea generation literature supports Samset and Volden’s 
(2016) paradox of the opportunity space and paradox of predict and provide. 
If an idea is not generated by an appropriate mechanism, then it is more likely 
to be a poor one. 

A project idea can emerge “from many different quarters such as a pro- 
spective funder, a business unit manager, a services department, an employee, 
an external consultant or even a supplier” (Zwikael & Smyrk 2019, p. 194). 
While the literature acknowledges the critical importance of idea generation 
or project ‘ideation’, the concept has been given little attention within the 
project management (Pinto & Winch 2016) and new product development 
literature (Joachim & Spieth 2020; Kock et al. 2015). According to Chenger 
and Woiceshyn (2021), the literature focuses on the evaluation and selection 
of projects, while ‘little is said’ about how new project concepts, opportu- 
nities or ideas are identified or generated during the front-end. Instead, the 
authors (p. 176) state it is widely assumed that “staff scout for opportunities, 
screen them through formal project management processes, and present those 
selected to the executive decision makers, who typically are not involved in 
generating project ideas” — an assumption that does not apply to projects in 
high-risk, high-reward contexts. Thus, project contexts such as the Defence 
may warrant their own investigation into the project idea identification and 
generation process. 


3.3.3 Business case development 


The development of a business case is the third project front-end activity we 
identified within the project management literature. A business case contains 
the information needed to assess whether a project is worth the investment of 
its funding organisation (Zwikael & Meredith 2019). Because it is not always 
appropriate for the person who came up with the original idea for a project 
to lead the business case development, this activity will often be driven by 
someone else, who is identified as the project ‘champion’ (Zwikael & Smyrk 
2019, p. 191). If the champion lacks the technical skills and time required to 
write the business case, someone else, such as the project-owner-designate 
or the project-manager-designate can be appointed to assist with that work. 
The business case includes information regarding the expected delivera- 
bles, estimated duration of project execution, its cost, cash flow, risks, com- 
munications strategy and governance structure. The benefits the project is 
expected to generate for its funding organisation and key stakeholders is an- 
other important piece of information included in the business case. As both 
the strategic and tactical performance of a project is evaluated to determine 
whether the project is a success, a project should be effective, relevant and 
sustainable (Samset and Volden, 2016) — it should deliver benefits and value. 
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The benefits and value a project is intended to provide need to be set in 
the business case. Zwikael et al. (2018) argue that project target benefits need 
to have three characteristics to be well-defined and effective in generating 
value: (1) specificity, for example, target benefits have a specific target value, 
(2) attainability, for example, the organisation has the capacity to realise the 
target benefits and (3) comprehensiveness, for example, target benefits reflect 
the views of key stakeholders. 


3.3.4 Business case appraisal 


The final project front-end activity we have identified within the literature is 
the appraisal of the business case. Once the business case has been assembled, 
it is presented to the potential project funder (the entity or entities with the 
authority to commit funds to the proposed project). The funder assesses the 
business case and determines one of three potential outcomes: (1) the busi- 
ness case is accepted and the project is funded, (2) the business case needs to 
be reworked and resubmitted or (3) the business case is rejected. Ultimately, 
appraisers (the project funder) assess the document to determine whether 
the project offers an appropriate trade-off between return and risk, and to 
assess how the project ranks against other projects competing for investment 
funds (Zwikael & Smyrk 2019). The project is the responsibility of the project 
owner, once the project has been formally approved (Williams et al. 2019; 
Zwikael & Meredith 2018). 


3.4 Methodology 


To investigate the logic of the front-end phase and address the research ques- 
tions, we adopt a qualitative approach. We selected this approach as our re- 
search investigates activities within the front-end phase that are not yet well 
understood, such as project trigger identification and project idea generation. 
Our research is guided by a critical realism ontological perspective in which 
an entity can exist without someone directly “observing, knowing and con- 
structing” it (Fleetwood 2005, p. 199). Instead, an entity can exist and be 
studied via its causal efficacy (Danermark et al. 2002). Qualitative methods 
have been used previously to study the front-end of projects, for example 
through interviews (Edkins et al. 2013), literature reviews (Williams et al. 
2019) and ‘cased studies’ (Samset & Volden 2016). 


3.4.1 Data collection 


We collected data using document analysis. Document analysis is a systemic 
procedure used to review or evaluate documents (Bowen 2009). As Merriam 
(1988, p. 118) states, document analysis can be used to “uncover meaning, 
develop understanding, and discover insights”. First, we identified publicly 
available Defence documents published for the purpose of explicating the 
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front-end phase of the ‘capability lifecycle’. We then applied content anal- 
ysis to identify and understand each of the steps in the front-end phase of a 
Defence project (Bowen 2009). As most Defence project related data is classi- 
fied, we developed a hypothetical Defence project case study to describe and 
explain the Defence front-end process in context. Once the steps had been 
identified, we compared them with the four front-end activities identified in 
the project management literature to ascertain the strengths and limitations 
of each. 


3.4.2 The Australian Defence Force case study — context 


Defence offers an excellent case study for other project industries to learn 
from, as it delivers large, complex acquisition and development projects, 
managed by highly trained professionals, in one of the most advanced project 
environments in the world. We also selected Defence for analysis as the front- 
end phases of large, complex projects are less well understood than those of 
small to medium-sized projects (Williams et al. 2019). In particular, our re- 
search focuses on the Defence’s Force Design Division (FDD). Force Design 
supports the Vice Chief of the Defence Force (VCDF) as the Joint Capability 
Authority responsible for determining the preparedness requirement for the 
Joint Force. This division was established as an outcome of the 2015 First 
Principles Review (FPR) for the purpose of producing a permanent Force 
Design function (Department of Defence 2015). 


3.5 Findings 


In this section, we first discus each of the steps Defence follows during a pro- 
ject front-end phase. We then use a hypothetical Defence project to demon- 
strate each of the steps in practice and define key Defence terms. Each of the 
steps in the Defence front-end process is then linked to the four front-end 
activities we identified in the literature (Table 3.1). 


3.5.1 The front-end phase of a Defence project 


The Defence front-end phase is designed to support Defence’s strategic 
objectives, which are “to shape Australia’s strategic environment; to deter 
actions against Australia’s interests; and to respond with credible military 
force, when required” (Department of Defence 2020b, p. 1). We viewed the 
front-end phase of a Defence project as being relevant for this chapter because 
it delivers large, complex acquisition and development projects, managed by 
highly trained professionals, in one of the most advanced project environ- 
ments. Therefore, we argue that the front-end project literature can benefit 
from analysing the Defence front-end phase. The Defence front-end process 
consists of 11 clearly defined and interdependent steps. These steps are dis- 
cussed next and illustrated with a hypothetical project example. Our analysis 
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is generalised where appropriate to enhance the applicability of our findings 
and conclusions to organisations outside Defence. 

The capability lifecycle (CLC) is the process used by Defence to link “stra- 
tegic direction, developing concepts, defining requirements, acquisition, 
introduction into service, sustainment, upgrade and disposal of major cap- 
ital assets (equipment, facilities and ICT)” (Department of Defence 2020a, 
p. 3). The cycle consists of four phases: (1) Strategy and Concepts, (2) Risk 
Mitigation and Requirements Setting, (3) Acquisition and (4) In-Service and 
Disposal. Using the front-end process discussed in the project management 
literature, the first two CLC phases constitute the front-end of a Defence pro- 
ject. As a result, a Defence project begins with a discussion around Strategy 
and Concepts and ends when government approval to commence the project 
is granted. A Defence project is defined as “a unique, transient endeavor, 
undertaken to achieve planned objectives” (Department of Defence 2020a, 
p. A-7). 

A complex Defence project needs to pass through three approval submis- 
sion gates — Gate 0 (Defence Investment Committee approval; occurs at the 
end of the first CLC phase), Gate 1 (government approval; occurs in the mid- 
dle of the second CLC phase), and Gate 2 (government approval; occurs at 
the end of the second CLC phase). All major projects (exceeding $20 million) 
require government approval. The Defence CLC’s gates are similar to the 
Norwegian Quality Assurance regime in the front-end of major public pro- 
jects (Samset & Volden 2016). However, the Norwegian model has two re- 
view gates, whereas the Australian model has three for very large projects. 
Further, whereas in Defence, one of the three gates is internal to Defence, 
the review gates in Norway are only external. The first Norwegian gate is 
called ‘Quality Assurance 1’ (QA1), when the conceptual solution goes to 
Cabinet for approval to commence work on a pre-project, and the second is 
called ‘Quality Assurance 2’ (QA2), when then project is submitted to the 
parliament for approval and funding. 

The front-end phase of a Defence project can be further broken down into 
11 steps: (1) identify change, (2) qualify risk, (3) prioritise risk, (4) develop 
options, (5) test options, (6) identify offsets, (7) test portfolio options, (8) 
SMART buyer framework, (9) contestability framework, (10) risk mitigation 
and requirements setting and (11) additional risk mitigation and requirements 
setting. ‘Options’ are referred to as project ideas or alternatives in the litera- 
ture. Steps 1 through to 9 constitute the first phase in the Defence front-end 
(‘Strategy and Concepts’ phase), while steps 10 and 11 make up the second 
phase (‘Risk Mitigation and Requirements Setting’ phase). During the sec- 
ond phase, risk is reduced and requirements are defined so that investments 
are defensible and a contract can be entered into. The first seven steps in the 
front-end form the Defence Capability Assessment Program (DCAP). FDD 
activities are conducted primarily through the DCAP. 

Next, we define key Defence terms, discuss each step in the Defence front- 
end process and provide a hypothetical case study to illustrate each of the 
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steps. Our hypothetical project involves the development of a jet, labelled the 
‘Blue Jet Project’. 

Step 1: Identify Change. The purpose of the first step in the front-end of 
a Defence project is to identify change through an assessment of the envi- 
ronment. This involves assessing internal and external inputs to the organi- 
sation. Inputs may include changes to strategic direction, technology, partner 
concepts and developments, identified gaps and opportunities, government 
and industry assessments, budget and deficiencies, as well as threats. Defence 
defines a threat as “an uncertain event, trend or condition that may result in 
negative outcomes” (Department of Defence 2021a). This step also identi- 
fies which Force Packages are affected by the change, to what extent and at 
what time. Force packages are “a mix of capabilities over a 20 year period” 
(Department of Defence 2020a, p. 16). A capability is defined as “The power 
to achieve a desired operational effect in a nominated environment within a 
specific time and to sustain that effect for a designated period” (Department 
of Defence 2020a, p. A-2). 

Blue Jet Project: A new intelligence report suggests a major threat — the 
development ofa military jet plane by another country that can fly faster than 
any military jet owned by Defence. 

Step 2: Qualify Risk. The second step involves conducting an impact as- 
sessment of the changes so as to qualify the risk in relation to Defence’s ability 
to achieve its strategic objectives. Risk is defined by Defence as “possible 
events that, if they occur, will impact on corporate goals and strategic objec- 
tives” (Department of Defence 2019, p. 22). Risk may be viewed negatively as 
a threat or positively as an opportunity. Risk is qualified and risk statements 
are developed using the Australian Defence Force Risk framework, which is 
based on likelihood and consequence (Department of Defence 2019). This 
step enables Defence to understand the accumulation of risk and opportunity 
to offset it. 

Blue Jet Project: It was determined that the risk posed by the faster jets and 
their ability to exceed existing and planned Defence jet capabilities was ex- 
tremely high. 

Step 3: Prioritise Risk. The third step involves a series of workshops to 
prioritise the risks and opportunities resulting from Step 2 before they are 
presented to senior decision-makers for consideration and direction. Opti- 
mising the effectiveness of the prioritisation and selection process is crucial as 
financial constraints prohibit all risks from being mitigated. 

Blue Jet Project: The risk posed by the faster jets was prioritised as greater 
than any other risk identified elsewhere and it was determined that this risk 
must be addressed. 

Step 4: Develop Options. Once the risk has been determined, Defence 
conducts a gap analysis between the current force and the future force. This 
leads to the generation of project options. These options may be to adjust 
resourcing levels to a Force Package, add new capabilities to a Force Package, 
substitute new Force Packages, or build new Force Packages. In addition, 
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the Services and other Groups independently identify, prioritise and select 
options they wish to develop to address their specific requirements. Once 
the Services and other Groups have identified the options they would like 
to be developed, and have independently completed their prioritisation and 
selection processes, the proposed options are submitted to FDD, which then 
works collaboratively with the Services to identify, prioritise and integrate 
new options that work across, or enable the Joint Force. The criteria used 
in Force Design’s prioritisation process have a whole of Defence rather than 
single Service focus. Each option is assessed in terms of cost and industry 
implications. 

Blue Jet Project: Three options were developed to respond to the risk posed 
by the faster jets, and to particular Force Packages. ‘Capability Option A’ pro- 
posed a new capability, as an air-based technology that would allow Defence 
to neutralise the risk posed by super-fast jets — specifically, the development 
of a new jet that can exceed the speed of the other country’s jets, thereby 
allowing our jets to shoot their jets down. When considering a capability 
option, Defence considers all of the associated Fundamental Inputs to Capa- 
bility (FIC). An FIC is defined as, 


a standardised set of nine resource and organisational enabling inputs, 
designed to enable the effective generation of Defence capabilities. The 
nine Fundamental Inputs to Capability are organisation, command and 
management, personnel, collective training, major systems, facilities and 
training areas, supplies, support and industry. 

(Department of Defence 2020a, p. A-4) 


In our case, a major system that makes up a part of Capability Option A is 
the Blue Jet Project on which this hypothetical case study is based. Additional 
FIC would include the recruitment of pilots, engineers and ground staff to 
operate, maintain and support the jets, and the training facilities to train the 
pilots to fly the new jets. Another option was to upgrade existing missile 
Defence systems against the threat the faster jets posed (Capability Option B), 
while a third option was to develop a new technology that would allow De- 
fence to remotely take control of the faster jets if they were to enter sovereign 
airspace (Capability Option C). 

Step 5: Test Options. The fifth step involves testing each capability option 
to assess its feasibility and value, and in particular how it would address the 
prioritised risks. The analysis is conducted, in part, using experimentation 
and war gaming. Each capability option is tested against Australian Contin- 
gency Context Scenarios (ACCS) which are derived from Defence’s strategic 
objectives (Department of Defence 2021b). 

Blue Jet Project: In this case, Capability A performed well during experi- 
mentation and war gaming. It was valued more highly than the alternative 
two options (Capability B and C) developed as potential responses to the risk 
posed to Defence by the faster jets. 
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Step 6: Identify Offsets. During this step, offsets that can be made within 
the existing budget to accommodate new or additional investments are iden- 
tified. This step also involves identifying the risks to the organisation should 
an existing project be rephrased/re-scoped or divested, given that each pro- 
ject within the budget addresses one or more risks. 

Blue Jet Project: A new Defence project that was about to be approved was 
identified as a potential candidate for re-structuring and rephrasing, which 
would free funds for the Blue Jet Project. 

Step 7. Test Portfolio Options. The seventh step involves identifying 
the net effect of proposed new investments offset in terms of the remain- 
ing risk resulting from the revised portfolios. Project portfolio management 
is a “dynamic activity through which an organization invests its resources 
to achieve its strategic objectives by identifying, categorizing, monitoring, 
evaluating, integrating, selecting, prioritizing, optimizing, balancing, au- 
thorizing, transitioning, controlling, and terminating portfolio components” 
(Project Management Institute 2017). 

Blue Jet Project: The portfolio options, which included Capability A (among 
existing capabilities), were tested against the remaining risk resulting from 
the revised portfolios. 

Step 8: SMART Buyer. The SMART buyer framework is an analysis and 
risk-based decision-making framework, which is facilitated by an Independ- 
ent Assurance Review (IAR) team. The purpose of this step is to understand 
the circumstances that led to the project and identify the ‘drivers’ or triggers. 
For example, there are a number of acquisitions drivers, such as needs, tech- 
nology, security and financial drivers. There are also sustainment drivers, 
such as support to operations, in-service considerations and commercial driv- 
ers. Once a ‘drivers profile’ has been developed, analysis and decision-making 
takes place to then produce tailored approval, asset management, governance 
and management, and commercial strategies for inclusion in the Project Exe- 
cution Strategy (PES) submitted at Gate 0 (Ryan & Soutberg 2020). SMART 
buyer workshops are conducted with key stakeholders during this step to 
“develop an improved understanding of a project’s risk profile at workshops 
facilitated by experienced and independent Defence personnel and industry 
experts” (Department of Defence 2017, p. 125). 

Blue Jet Project: The proposal for the Blue Jet project was submitted to 
SMART buyer for review. The proposal was included in a risk and drivers 
workshop, strategy development workshop and ‘red team review’ before re- 
ceiving approval from the Delivery Group Division or branch head to proceed 
to the next step. The SMART buyer team provided an improved understand- 
ing of the Blue Jet Project’s risk and drivers, and developed tailored strategies. 

Step 9: Contestability. During this step, Contestability Division checks 
if the proposal aligns with strategic and resource guidance, and if it can be 
implemented as proposed. In addition, Contestability Division checks to see 
if the risk assessments and treatment strategies are appropriate, if the basis for 
the decisions made is acceptable, and if the cost and schedule estimates are 
acceptable. The SMART buyer framework and the contestability framework 
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are applied three times during the front-end phase at Gates 0, 1 and 2 (Ryan 
& Soutberg 2020). 

Blue Jet Project: Contestability Division reviewed the Blue Jet Project pro- 
posal against the criteria listed above. As the proposal was found to align 
with strategic and resource guidance, have appropriate related assessments 
and strategies, be the basis for decisions, have acceptable cost and schedule 
estimates, and confirmed that the project could be implemented as proposed, 
Contestability Division approved the project for consideration at the next 
Gate. 

Gate 0. The portfolio options are submitted to the Defence Investment 
Committee for consideration. If approved, the selected portfolio receives pre- 
Gate 1 funding. 

Step 10: Risk Mitigation and Requirements Setting. Risk reduction ac- 
tivities include modelling and simulation, commercial risk assessments, risk 
reduction studies and trade-off studies. The purpose of the requirements set- 
ting phase is to define exactly what is to be acquired to provide a clear basis 
for public expense, the requirement to enter into a contract and to confirm 
that what has been delivered is acceptable. 

Blue Jet Project: During this step, risks associated with the Blue Jet Project 
would be identified and reduced where possible using the risk reduction ac- 
tivities stated above. For the Blue Jet Project, the risks may include insuffi- 
cient pilots to fly the jets, no instructors within Australia to train the pilots 
and/or no available space to store the jets when not in use. When setting 
requirements, the specific requirements for the Blue Jet Project would be 
identified, such as the number of jets and trained pilots required, the location 
the jets would need to be stored at when not in use, the storage and mainte- 
nance requirements for the jets, and the amount of fuel the jets would require 
to operate for a given period. 

Gate 1. The project is submitted to government for consideration. If ap- 
proved, the project will receive pre-Gate 2 funding. 

Step 11: Additional Risk Mitigation and Requirements Setting. Additional 
risk mitigation and requirement setting activities take place after Gate 1 
and prior to Gate 2. These include further requirements setting (moving 
toward contract ready specifications), commercial risk assessments and trade- 
off studies. Funds are committed incrementally during Step 10 and 11 in 
relation to the degree of risk and uncertainty, as well as the definition of the 
requirements. 

Blue Jet Project: Further requirements setting, commercial risk assessments 
and trade-off studies were conducted on the Blue Jet Project. Funds were in- 
crementally released to the project during the Risk Mitigation and Require- 
ments Setting phase in accordance with the degree of risk and uncertainty, 
and definition of the project requirements. 

Gate 2. The project is submitted to government for consideration. If ap- 
proved, the project will receive funding for acquisition. The front-end phase 
then concludes, and responsibility is handed over to the Capability Acquisi- 
tion and Sustainment Group (CASG) to deliver the approved project. 
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3.5.2 A comparison of the Defence process with the 
project management literature 


Table 3.2 shows how each of Defence’s 11 steps detailed above fit within the 
four front-end activities we identified within the literature (Table 3.1). This 
analysis also shows how the Defence process may contribute to the project 
management front-end literature by expanding and enriching each of the 
four activities. The Defence front-end steps are included in Table 3.2 below, 
along with a brief description of the associated front-end activity identified 
within the literature. The Defence front-end phase could significantly enrich 
and expand the front-end activities identified in the literature, and provide a 
framework for effective front-end practice. 


3.6 Discussion 


The front-end of a project is a decision process. This decision process “com- 
mences with the identification of a stimulus for action and ends with the 
specific commitment to action” (Mintzberg et al. 1976, p. 246). This chapter 
explores the front-end phase of a project. We reviewed the relevant literature, 
conducted a qualitative investigation involving document analysis to reveal 
the front-end phase of Defence projects and presented our findings using a 
hypothetical case study. Defence presents an ideal case study for the project 
management literature, as the front-end of a Defence project is rigorous, 
comprehensive, protracted, advanced and well-documented. We then com- 
pared knowledge of the front-end project phase from the project manage- 
ment literature with the Defence front-end project phase. 

Our findings and their implications are discussed in the following four 
sub-sections. In Section 3.6.1 we address research question 1 (How should an 
effective project front-end phase be constructed?). In Section 3.6.2 we respond to 
research question 2 (How are good project ideas generated?). Next, we address 
research question 3 (How can effective front-end practice contribute to the resolution 
of common front-end paradoxes?) in Section 3.6.3. Our implications for project 
management practice are then discussed in Section 3.6.4. 


3.6.1 An effective project front-end phase 


In this section, we draw on the findings presented in Section 3.5.1 to present 
an effective project front-end phase that may assist project management prac- 
titioners to construct the front-end phase of a project. This phase is mapped 
in Figure 3.1. It expands each of the four activities identified in the literature 
(see Table 3.1) into 12 separate steps, which include: (1) two businesses cases — 
an initial business case and a detailed business case — rather than the one 
referred to in the literature, (2) two approval gates — a conceptual approval 
gate (following the initial business case) and a funding approval gate (follow- 
ing the detailed business case) — as opposed to the one recommended in the 
literature, and (3) two detailed processes for front-end activities that are not 
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Table 3.2 A comparison of Defence and project management literature front-end 


processes 

Defence Front-End Steps Description Project Management 
Literature Front-End 
Activities (see Table 3.1) 

1. Identify change Identify change in the Activity 1 — Project 


environment and determine trigger identification 
how the Force Packages are 
affected by the change 
2. Qualify risk Develop risk statements for 
each of the identified relevant 
changes using the Australian 
Defence Force risk framework 
3. Prioritise risk Workshop to prioritise risks 
4. Develop options Build Force Package options sets Activity 2 — Project idea 
by increasing resources, adding generation 
new capabilities, substituting 
Force Packages or building 
new Force Packages 
5. Test options Test options to confirm how they 
address risk and identify best 


value for money treatment 


options 
6. Identify offsets Identify offset strategies for new Activity 3 — Business 
investments case development 
7. Test portfolio options Test options to confirm net 
positive impact 
8. SMART buyer Analysis and risk based decision- 
making framework 
9. Contestability Check proposal aligns with 


strategic guidance and 
resource guidance, and can be 
implemented as proposed 


Defence Investment Committee Decision (Gate 0) 
10. Risk mitigation and During this stage, risk is reduced 
requirements setting and requirements are defined 
so that investments are 
defensible and a contract can 
be entered into 
Government Decision (Gate 1) 


11. Additional risk Additional risk mitigation and 
mitigation and requirement setting activities 
requirements setting 
Government Decision (Gate 2) Activity 4— Business 


case appraisal 


clear in the project management literature — the project trigger identification 
process and the project idea generation process. We have identified eight 
implications for practice stemming from the recommended front-end process 
shown in Figure 3.1. These implications are discussed in Section 3.6.4. 
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Activity 1: Project trigger identification Activity 2: Project idea generation 


2. Conduct an 3. Prioritise risks r 
A | 4. Generate |,| 5. Test project 
nd impact and HP panes ‘ 
project ideas ideas 


1. Assess the 
environment 


assessment opportunities 


+ 


Activity 3: Business case development 


at, al g a 10. Risk 
8. Proposal 9. Gate 0 | i 


6. Identify 7. Test portfolio lit Submit initial mitigation and 
offsets options AUS aeons requirements 
assurance business case RS 
setting 


Activity 4: Business case appraisal 


11. Proposal 12. Gate 1 
quality Submit detailed 
assurance business case | | 


Figure 3.1 The proposed project management front-end process. 


Earlier research has identified a lack of a structured framework, unclear 
definition of roles and activities, and ineffective communication channels as 
major challenges facing the front-end planning process (Oh et al. 2016). By 
mapping the front-end of a project, this research assists people involved with 
the front-end phase to overcome these challenges by providing guidance as 
to which activities should occur and when. This would enable the assignment 
of roles with clear accountabilities and responsibilities, and support the con- 
struction of communication channels. 


3.6.2 Project idea generation 


This section discusses the findings presented in Section 3.5.1 in order to re- 
spond to our second research question — How are good project ideas generated? In 
Defence, project options are developed and tested by involving each of the 
seven steps in the DCAP. However, we found that in Defence, project op- 
tions are primarily developed through a five-step process: (1) identify change, 
(2) qualify risk, (3) prioritise risk, (4) develop options and (5) test options. 
Each step builds on the one that precedes it. 

The front-end of a project is triggered by the identification of a problem or 
opportunity by the project funder (Zwikael & Smyrk 2019), although how ex- 
actly this trigger should be identified to enhance the effectiveness of project 
idea generation is unclear. In Defence, project triggers are identified through 
an assessment of change in the environment at Step 1. The environment 
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consists of all internal and external inputs to the organisation, such as changes 
in strategic direction, technology, partner concepts and developments, iden- 
tified gaps and opportunities, government and industry assessments, budget, 
deficiencies and threats. 

A new project idea should be guided by the anticipated effect rather than 
the present undesired situation (Williams & Samset 2010). Thus, the trigger 
for the generation of the idea (Step 1 — identify change) is important, as it 
determines the anticipated effect (Step 4 — develop options). The anticipated 
effect may be viewed through the lens of the “Now-No-Yes’ (2NY) scenario 
map (Zwikael & Smyrk 2019, p. 27), where three alternative states are com- 
pared. First, the ‘Now’ scenario, which is described as “the way the world is 
shaped today in terms of selected variables of interest” is analysed in the De- 
fence process in Step 1 (identify change). This step analyses the current state 
and how various changes may impact it. Second, the ‘No’ scenario is “the 
way the world will be shaped in the absence of the project” (Step 2 — qualify 
risk), that is, the state Defence will be in if the threat is materialised without 
an appropriate response. Third, the ‘Yes’ scenario, which is “the way the 
funder wants the world to be shaped as a result of the project” is analysed as 
the anticipated effect (Step 4 — develop options), that is, the state Defence will 
be in if they respond to the threat through the development of a new project. 

Good ideas are generated when a decision-making framework is followed 
that ensures the options generated and tested are clearly linked to the initial 
trigger for the project (Step 1 — identify change), are motivated by a qual- 
itative and quantitative understanding of the risk to the organisation (Step 
2 — quality risk), have been prioritised against other risks and opportunities 
that can impact the organisation (Step 3 — prioritise risk) and, notably, have 
had not one, but a wide range of alternative ideas developed (Step 4 — develop 
options). It is then important to test the option to confirm how it will address 
the risk and fit within the organisation’s wider project portfolio(s) (Step 5 — 
test options). 

Engaging in an extended project idea generation decision-making frame- 
work such as this one can reduce the likelihood of various cognitive factors 
impacting project idea generation such as cognitive biases, lock-in, path de- 
pendence, political pressure and strategic misrepresentation, and increase the 
likelihood of generating a good project idea. The Defence’s five-step project 
options development and testing process also helps to ensure the project trig- 
ger (step 1 — identify change) links to the organisation’s strategic objectives 
(step 2 — qualify risk) and to the project outcomes. These linkages support the 
project proposal by providing evidence of the value of the project. This, in 
turn, should positively impact project investment decision quality. 


3.6.3 Resolving common front-end paradoxes 


This section further analyses the findings presented in Section 3.5.1 in re- 
sponse to our third research question — How can effective front-end practice con- 
tribute to the resolution of common front-end paradoxes? The Defence front-end 
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phase may contribute towards the resolution of the four common front-end 
paradoxes discussed in this chapter. The occurrence of the first paradox, ‘the 
paradox of the significance of front-end management’, may be lessened by the 
findings presented in Section 3.5.1, as this section contributes to project man- 
agement knowledge by detailing how an effective front-end phase should 
be constructed, considering the larger context in which the project is “ide- 
alized, validated, and shaped” (Pinto & Winch 2016, p. 238). This reduces 
the uncertainty typically associated with the front-end (Kolltveit et al. 2004), 
reduces the risk of unnecessary investment in the front-end (Morgan 1987) 
and ensures that the front-end is not overlooked due a lack of knowledge — 
what the objectives of the front-end are, and how they can best be achieved 
(Pinto & Winch 2016). 

The findings presented in Section 3.5.1 may also contribute to the resolu- 
tion of ‘the paradox of the opportunity space’. Specifically, the detailed, five- 
step project ideas development and testing process increases the likelihood that 
multiple project ideas will be generated based on a specific relevant change in 
the environment, rigorous assessment of the impact of the change to strate- 
gic objectives, and prioritisation of the impact before undergoing testing and 
experimentation. Therefore, it is less likely that the project idea eventually 
approved will be as a result of the factors that may explain the paradox of the 
opportunity space, such as bounded rationality, political pre-determination, 
experiences and preferences of the decision-makers or the tendency of disci- 
pline experts to focus on particular aspects while overlooking others. 

Section 3.5.1 may also have implications for another common front-end 
paradox — The paradox of ‘predict and provide’. There is a tendency for 
planners to select the most obvious solution. If usage is exceeding capacity, 
increase the capacity. This is perhaps the easiest and most straightforward 
response to a problem. This paradox is attributed, in part, to a decoupling 
of needs and benefits analyses from the most pressing political priorities and 
goals. The Defence front-end process supports linkages between the original 
trigger for a project, strategic objectives and the approved project. Specifi- 
cally, these linkages are supported by following a clearly defined sequence of 
steps that ensures each stage of a project’s development is logically connected 
to the preceding step. The solution can then be directly linked to the prob- 
lem and strategic objectives. Furthermore, the Defence process involves a 
step where alternative project ideas should be developed. Options should be 
explored by a work group consisting of individuals with expertise in the area, 
but also with an understanding of the organisation’s existing project portfo- 
lio/s and knowledge of the organisation’s strategic objectives, and the broader 
political and financial landscape. 

Finally, the Defence front-end process involves an extensive, rigorous qual- 
ity assurance check by an independent group of experienced experts to ensure 
the linkages between the project trigger, strategic objectives and proposed 
project solution are clearly defined and supported with evidence. This occurs 
before the proposed project is submitted for consideration at Gates 0, 1 and 2. 
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The fourth and final front-end paradox that we believe our findings have 
implications for is the ‘paradox of strategic alignment’. The Defence front- 
end process can contribute to the resolution of this paradox by ensuring the 
objectives and benefits of a project are clearly articulated in key front-end 
documentation using effective frameworks. The Defence process can also 
help to ensure a project idea is aligned with strategic objectives by including 
a step that involves testing an idea against a scenario which reflects Defence’s 
strategic objectives. Finally, an explanation for the paradox of strategic align- 
ment may be that projects often lack shared objectives and agreement on 
the objectives, and have too many goals, or too many unrealistic or overly 
ambitious goals. This problem is largely avoided by Defence, as the front-end 
process is mostly carried out by FDD — a group who sits between the services 
and government to ensure the project ideas sent to government for considera- 
tion reflect the objectives of the whole of Defence, rather than one particular 
service. This group shares an understanding of the business objectives as well 
as commitment towards business outcomes and the plans for achieving them. 


3.6.4 Implications for project management practice 


Our research has several implications for project management practice. These 
implications have been presented in Table 3.3. This table provides the front- 
end activity identified in the literature, a limitation in the literature associ- 
ated with this activity, relevant findings from the Defence case study and the 
implications of these findings for project management practice. 


3.7 Conclusion 


The project front-end is considered unclear or ‘fuzzy’. Little is known within 
the project management literature and bodies of knowledge about the process 
organisations should follow to achieve the objectives of the front-end. To 
respond to this gap in the literature, we conducted a qualitative investigation 
using document analysis into what is widely considered to be an advanced 
and effective project front-end practice — the front-end of Defence’s capa- 
bility development lifecycle. We used a hypothetical case study to present 
our findings, and then compared the Defence project front-end with the 
project management literature to enhance research knowledge in this area. 
Finally, we explored common paradoxes associated with the front-end phase 
and considered how the Defence project front-end process may contribute 
towards the resolution of these paradoxes. 

Our research contributes to project management practice by enriching 
and extending the front-end phase with important insights from Defence. 
Specifically, the implications are for project management practitioners, as- 
sisting them to construct an effective front-end phase that will facilitate the 
achievement of front-end objectives. Implications are also for the literature 
and practice in providing suggestions as to how common front-end paradoxes 
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may be resolved. Finally, implications are for organisations, assisting them 
to achieve a competitive advantage through the generation of better project 
ideas than their rivals. 
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4 Undertaking the project 
front-end 


Monique Aubry and Serghei Floricel 


4.1 Introduction 


Over the last two decades, several scholars have shown the critical impor- 
tance of front-end activities for project success, but have also pointed to the 
irregular, intractable nature of these activities (Miller & Lessard 2001; Sam- 
set & Volden 2016a; Williams et al. 2019). The results raise doubts about 
the dominant prescriptive models for front-end activities, which depict them 
as a hierarchical sequence of rational decisions (e.g. Kerzner 2017). Similar 
prescriptions have already been implemented in the governance frameworks 
of countries such as Norway and the UK, despite a lack of theoretical and 
empirical grounding, and limited ex post validation of underlying models 
(Ahlemann et al. 2013; Klakegg et al. 2008; O’Leary & Williams 2008; 
Volden 2018). Based on a process-based theoretical perspective, and using sev- 
eral empirical vignettes from projects currently being developed in Quebec 
(Canada), this chapter attempts to shed new light on the front-end difficulties 
of major projects. While we believe that prescriptive approaches provide an 
invaluable project analysis and development tool, we want to give those con- 
sidering front-end activities an understanding of why these prescriptions are 
seldom implemented as intended. In particular, we present reasons for two of 
the paradoxes highlighted by Samset and Volden (2016a), in which time plays 
a subtle but crucial role. The first, ‘the paradox of the significance of front- 
end management’ (Paradox 2), contrasts the critical importance of the front- 
end with the limited resources dedicated to this stage of the project relative 
to the execution stage, while the second, ‘the paradox of the opportunity 
space’ (Paradox 4), deplores the selection of conceptual solutions without first 
systematically exploring the opportunity space. 

In prescriptive models inspired by decision theory (Keeney et al. 1993; 
Savage 1972) and other general normative theories, such as rational systems 
design (Pahl & Beitz 2013; Suh 1998), front-end processes remain outside 
time, based on the implicit assumption that their own dynamics should not 
impact results. With these models, time is a concern only for the ability to 
foresee future evolutions, as longer term predictions are typically deemed 
more uncertain, and to account, for example through Bayesian probability 
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updating, for incoming information, such as the results of drilling on a pro- 
spective site (Caron et al. 2016; Fye et al. 2013). With these exceptions, time 
should have no impact on front-end evaluation outcomes. The temporal un- 
folding of the process itself, for example its timing, duration or stoppages, 
should not affect results apart from the interval required for knowledge pro- 
duction activities and uncertain events to take place. Yet studies of megapro- 
jects’ front-end inspired by political science, including Bert van Wee’s study 
in Chapter 6, suggest that the moment of, say, path announcement relative 
to the political calendar, especially in light of previous promises, may have 
a significant impact on decision-makers’ assessments and choices. Moreover, 
announcements are events in themselves, which, contrary to what norma- 
tive models suggest, make a difference in the world, and may change the 
perceived value of the project (Neelawala et al. 2015). But even stoppage 
periods, in which no activity is performed and no relevant events occur, seem 
to have a significant impact on the understanding and evaluation of projects 
(Dimitriou et al. 2013). The process itself appears to have its own temporal 
logics, beyond the time it takes to produce knowledge and to let relevant 
uncertainties play out. 

To advance our understanding of front-end dynamics, as a process situated 
in time and having its own temporal logics, we combine the view of projects 
as a phased decision process (Samset & Volden 2016a) with a social prac- 
tice perspective on the complex activities occurring in this front-end stage 
(Floricel et al. 2014). The latter perspective suggests that any seemingly stable 
feature of the world is achieved through many diverse and sustained efforts 
of the interested actors (Nicolini 2013). This means, for example, that a pro- 
ject in development is not just a solid best option waiting to be selected but 
a fragile, temporary outcome of multiple, often uncoordinated and perhaps 
contradictory efforts (Floricel & Piperca 2016). These efforts are themselves 
situated in time, and intertwined with multiple material and social processes, 
among which is the development of various objects, particularly of project 
representations (Nicolini 2013). Because prescriptive models put project rep- 
resentations at their core, particularly alternatives or solutions, assessments 
of their expected benefits and costs in various scenarios of future evolutions, 
as well as probability distributions over these possible outcomes, our analysis 
of the front-end process will focus on the temporally situated practices that 
develop and use representations of the future project, of its context and of its 
implementation. 


4.2 On representations 


From our perspective, a representation is a perceptual, conceptual and social 
construction that uses words, signs and drawings to refer to and relate in a 
particular way selected elements of the project and of the surrounding world 
(Floricel et al. 2011). For major projects, representations include a wide range 
of interrelated objects, such as pathway maps, concept descriptions, artistic 
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renderings, technical drawings, cost/benefit assessments, implementation 
plans, contracts, organisational charts, financial proformas and so on. Tech- 
nical design scholars depict the process as the gradual cognitive construction 
of a representation of the objects to be executed (Visser 2006). The logic of 
this construction appears to favour an evolution from abstract and schematic 
forms, such as flowcharts to increasingly detailed and concrete depictions, 
such as 3D CAD drawings (Chandrasegaran et al. 2013). One way construc- 
tion activities are situated in time is that advances rely upon and embed in 
their outputs the accumulating knowledge about the object and its context 
(Floricel et al. 2011). However, unlike rational prescriptions for systems de- 
sign, and even more than bounded rationality models with their satisficing, a 
practice perspective considers that “processes are always ‘emergent’ and fol- 
low multiple logics” (Lanzara 1999, p. 335). 

One logic refers to the temporal vagaries with which evidence comes to 
the attention of designers and to the improvisational, bricolage-like nature 
of representation construction. Because there is no rule for translating an 
abstract diagram into concrete forms, the process requires creative jumps 
which depend on what evidence seizes first and most forcefully, the indi- 
vidual and collective attention, which available concrete solutions capture 
the imagination at any given time, how ensembles fit together, and so on 
(Garud & Karnoe 2003; Miner et al. 2001). The practice view goes one step 
beyond the logics of knowledge production and its relation to representation 
construction by looking at representations as concretely captured on exter- 
nal supports, from paper to computer screens, as tools and even as front-end 
actors in their own right (Blomquist et al. 2010). The speed of representa- 
tion construction, including the ability to integrate various perspectives, and 
also the ease of subsequent changes depends on the material nature and the 
affordances of these supports. Paper sketches and simple mock-ups are easier 
to build and give more flexibility for a broad solution exploration, while 
sophisticated technologies such as Building Information Modelling (BIM) 
require additional preparation, but enable a more precise evaluation of design 
decisions (Boland et al. 2007). Representations also differ in the potential to 
become focal points for individual cognition, integrating the inputs of par- 
ticipants with different disciplinary backgrounds (Ewenstein & Whyte 2009; 
Knorr Cetina 1997). 

In addition, external representations are a key instrument for communi- 
cating about the project and eliciting assessments from decision-makers and 
stakeholders. But contrary to economic and political science models depicting 
these actors as rational and calculative, perhaps evaluating the project in light 
of alternative, political markets, practice-centred views consider a broader 
range of cognitive, emotional and social reactions triggered by representa- 
tions (Floricel & Brunet 2019). Some properties of representations enable pro- 
ject misrepresentation, while reactions include the escalation of commitment 
(Winch 2013). Yet, while some reactions appear irrational in light of norma- 
tive models, advances in neuroscience suggest, for example, that emotions are 
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essential in making effective decisions in a complex world (Hodgkinson & 
Healey 2011). In other words, the very possibility of cold, calculated decisions 
may be just another rational myth, and following this approach may even be 
counterproductive in some conditions. From this viewpoint, representations 
and the reactions they trigger may be an effective way to probe and decide 
on projects. Moreover, external representations may not only depict and help 
evaluate the project and its context, but by illuminating and constraining the 
project and the surrounding world in a particular way, they would also create 
a particular future (Comi & Whyte 2018). The material and visual turn in 
organisation science even suggests (Boxenbaum et al. 2018) that they inspire 
and enable actors in a non-trivial way to schedule activities, organise efforts 
and resource commitments and allocate responsibilities and risks. 

This brief review suggests that representation construction has its own 
particular dynamics, influenced by multiple inherent temporal logics, and 
by tensions between production activities and representation uses, as well as 
between the evolving interests of various actors and organisations (Smith & 
Lewis 2011). Focusing on the construction of representations and their roles 
sheds new light on the dynamics of front-end, particularly on the forces that 
push towards and against engaging in and deepening the solution explora- 
tion and evaluation effort. In our view, a focus on representations from a 
practice perspective helps us to grasp why front-end activities are rarely a 
linear unfolding process, why they go through several stop-and-start peri- 
ods or follow iterative sequences, and why ‘project history’ leaves a mixed 
legacy for implementation activities and the final project. The next section 
discusses four trade-offs that emerge around processes of representations con- 
struction, as seen from a practice perspective. Trade-offs are a privileged lo- 
cus for studying paradoxes, and analysing them advances our understanding 
of the two front-end paradoxes mentioned at the start of this introduction. 
Each trade-off is illustrated by two vignettes (numbered one to eight) from 
the recently completed or ongoing front-end activities of major projects, all 
in the province of Quebec (Canada). Data were obtained from public publi- 
cations, mainly project websites, newspapers and governmental publications. 
The chapter ends with lessons for those undertaking front-end activities and 
creating governance frameworks for major projects. 


4.3 Four representation-based trade-offs in the front-end 


Governance frameworks and organising practices recognise that developing a 
project representation is a gradual process, notably by tying in key decisions 
with respect to preliminary approval, full commitment and contractor selec- 
tion to specific, increasingly higher percentages of completion of the project 
design and engineering. However, these prescriptions do not fully account for 
the fact that drawings and other project representations are also tools used in 
social practices as epistemic objects, symbolic artefacts, future-making tools 
and boundary objects. Considering these concurrent roles, representations 
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become an excellent way of studying the complexity of interactions in the 
project front-end, suggesting the following four trade-offs. 


4.3.1 Knowing versus exploring trade-off 


This trade-off refers to the compromise between exploring a broad variety of 
opportunities, options and solutions, and producing reliable knowledge re- 
garding the potential of each of these options. Knowledge production about 
a project is conditioned by its definition, which goes hand in hand with the 
elaboration of increasingly detailed representations of the project and its con- 
text. Without defining what objects the project intends to execute, exactly 
where, when and how, developers cannot produce the concrete knowledge 
needed for a precise and reliable evaluation of its perspectives. The lack of 
concrete knowledge forces evaluators to rely on generic knowledge from 
past similar projects, which only supports approximate evaluations, for ex- 
ample through parametric estimation methods (Karaca et al. 2020). The 
relation between representation development and knowledge production is 
also influenced by several other logics. Specific uncertainties requiring ad- 
ditional knowledge production — known unknowns, so to speak — only start 
to be understood specifically as the project gets defined; before producing a 
representation of the particular form, technical solution or implementation 
mode of a project, the issue is, rather, ambiguity (Floricel & Miller 2001; Pich 
et al. 2002). Project definition and representation progress as new knowledge 
about these uncertainties comes in (Morris 2013), but, in line with the prac- 
tice view, the evolving project representation is an epistemic object, whose 
particular form conditions the specific ways in which participants with var- 
ious disciplinary and organisational backgrounds integrate their knowledge, 
understand the project and identify areas for which incomplete knowledge 
requires additional studies and tests (Ewenstein & Whyte 2009). In turn, rep- 
resentation production processes are temporally situated, emergent and im- 
provisational, conditioned by the affordances of material supports, arbitrary 
bursts of creativity, by the length of gestation periods for reconceptualisation, 
and the particular sequence of serendipitous encounters with actors, ideas, 
information, concrete forms and so on. While the project representation may 
go through several iterations, some of which bring significant changes, the 
process is also increasingly path dependent, as understanding and debates 
centre more and more around one core project option (Hellstrôm et al. 2013). 
Accumulated concrete knowledge is generally bound to a specific solution for 
which it was produced and cannot be redeployed for evaluating alternative 
solutions. 

This trade-off provides a direct account for Paradox 4, namely, selecting 
a conceptual solution without a systematic exploration of the opportunity 
space. This space does not sit out there waiting to be explored; in addition 
to imagining opportunities, in itself a serendipitous process, concrete rep- 
resentations have to be carefully and painstakingly built for every opportunity 
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that is seriously considered. Without a representation that reveals unknowns 
and complexities and guides specific knowledge production, evaluations are 
highly uncertain. Also, the produced knowledge is solution-specific. There- 
fore, any additional option considering calls to restart the representation and 
knowledge production processes means significant delays to investigate an 
opportunity whose value is very uncertain, given the scarce knowledge avail- 
able about it. This trade-off also provides an indirect account for Paradox 2, 
namely, the low investment in the front-end relative to execution. Front-end 
actors may understand the tenuous nature of knowledge production processes, 
and its dependence on the emergent and temporally situated representation 
development process, which means that the eventually selected option may 
not be any better than others abandoned earlier. They also understand that 
deepening the front-end study and evaluation may be useless, as some uncer- 
tainties cannot be reduced prior to execution, for example prior to digging 
on a particular site, which may first require the acquisition of the respec- 
tive rights which, in turn, may be conditioned by the approval of a project 
solution. Therefore, front-end decision-makers may prefer to quickly select 
a solution based on ‘secondary’ factors such as political attractiveness and 
regulatory compliance, and then make sure it is developed and implemented 
in the best possible way (Doloi 2011). This way of acting may be particularly 
attractive in conditions of project complexity and pressure to accelerate pro- 
ject delivery. 


Vignette 1: A swift decision-making process on 
reusing existent infrastructure 


The first vignette refers to the first phase of a major urban public trans- 
portation system project, which aims to give Montrealers modern, 
rapid and accessible public transport: the Réseau Express Métropolitain 
(REM). This automated light rail system (the first one in Quebec), 67 
kilometres long, with 26 stations, links the downtown, the airport and 
the Greater Montreal area. New lines complement and are integrated 
with the existing transportation systems as well as with sustainable 
mobility partners. At the moment of writing, the estimated cost is 6.3 
billion CAD (700 million over the initial estimation). Feasibility stud- 
ies were initiated in 2015, and construction started in 2018. Progressive 
opening is expected to begin in 2024 (two years later than the initial 
expected operation date). 

Outside the technical aspects, a major characteristic of this project 
is its financial model and governance approach (Schepper 2017). As 
in many other countries, infrastructure investments in Quebec have 
significantly increased over the last decades, mainly due to ageing 
infrastructures and fundamental changes in society. After the PPP 
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experiments, the Quebec Government (as many other governments) 
was searching for alternative financial solutions, allowing private funds 
to contribute to the renewal and construction of new public infrastruc- 
tures in conjunction with public interests. In this context, the Caisse de 
Dépôt et Placement du Québec — Infra (CDPQ Infra) was created in 
2017 by a special law in Quebec legislation (Gouvernement du Québec 
2017a). The realisation mode was defined as an innovative model based 
on the direct involvement of CDPQ Infra in all phases of the project, 
from the initial idea to the operation, and not only as a funding partner: 
“CDPQ Infra acts as a principal contractor for major infrastructure 
projects. We are responsible for all phases of a project: planning, financ- 
ing, execution and operation” (CDPQ Infra 2021). 

We found in the design phase of this project a good illustration of the 
trade-off to be made between knowing and exploring regarding the 
selection of path for the light rail section linking downtown to a north- 
west suburb, Deux-Montagnes. The design team quickly zoomed in on 
reusing existing infrastructure, mainly because this avoided politically 
sensitive expropriation, but also because it suited its strategy to acceler- 
ate the delivery of infrastructure. 

Interestingly, this section of the line includes a 100-year-old tun- 
nel under the Mont-Royal. The tunnel was constructed between 1912 
and 1918, and was highly innovative at the time. The tunnel is two- 
way, approximatively 5 kilometres long and 180 metres deep. It was the 
property of the national train company, the Canadian National (Ville 
de Montréal 2021) and, as of May 2020, was used for passengers only 
from the north-east and north-west suburbs to downtown Montreal. 
A well-known problem with the existing tunnel is its uniquely small 
ventilation chimney (Montréal 2021). This is the reason why only elec- 
tric locomotives were used in the tunnel. It is not surprising that part of 
the REM project included tunnel modernisation. 

This solution of reusing existing infrastructures has three major im- 
pacts. First, it prevents thousands of passengers commuting every day 
between home and work. The service stopped in May 2020 and is 
planned to reopen with the REM in 2024. Meanwhile, a bus service 
has been put in place, with a much longer travelling time (BAPE 2016). 

Second, there are security impacts (REM 2020). An explosion oc- 
curred only a few weeks after work started in the tunnel: old explo- 
sives from the time the tunnel was built were probably touched and 
exploded. Also, intrusive inspections revealed that the structural deg- 
radation of the tunnel was much more significant than anticipated due 
to the use of de-icing salt in a street above it. Salt infiltrations, year after 
year, due to waterproofing problems, have corroded the tunnel vault. 
At this point, several options were identified to correct the situation, 
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both for the tunnel and for the street infrastructure. The works will 
be realised from the interior of the tunnel to minimise the impacts on 
circulation. 

Finally, the agreement signed between the government and CDPQ 
Infra confirmed the exclusive utilisation of this section of the line by 
CDPQ Infra which prevents other opportunities (ARTM 2018) such 
as the mooted highspeed train between Montreal and Quebec City, 
which is now almost impossible (Schepper 2017). Yet the impact will 
have long-term consequences on transportation development. 

In this case, despite daily impacts on thousands of users in the short 
term, and the longer-term impact on future opportunities, the ‘know- 
ing’ part of this trade-off clearly prevailed over the exploration part. In- 
deed, project definition and representation development moved swiftly 
to knowing without investing effort and resources to explore alter- 
native representations of the project (Paradox 2) as confirmed in the 
environmental assessment: “The promoter did not perform a compar- 
ative assessment of various options allowing to improve services on the 
Deux-Montagnes suburban train line” (BAPE 2016, p. 9). The oppor- 
tunity space was not systematically explored as the choice of reusing the 
existing infrastructure was made, resulting in a firm solid representa- 
tion that will be difficult to change or adapt (Paradox 4). 


Vignette 2: The invisible nature of the underground 
infrastructure 


Contrary to the first vignette, where a rapid convergence towards a 
solution leads to a rather positive result (as of now) in terms of ad- 
vancing through the front-end towards execution, this second vignette 
describes a major problem that occurred with a municipal sewage col- 
lector dating back to 1930, right after site preparation works (Bisson 
2014). This situation led to a two-year delay in completion and cost 
overrun of 60 million CAD. 

Turcot Interchange constitutes a central mobility node in the Mon- 
treal area, used by 300,000 vehicles a day (Gouvernement du Québec 
2021). The new infrastructure replaces the previous one, built in the 
1960s, and described as an immense road spaghetti squeezed between 
train tracks, the St. Lawrence River (Canal Lachine) and residential 
areas. The new infrastructure includes four interchanges, three bridges, 
145 kilometres of roads, 31.5 hectares of new green surface and 8.1 
kilometres of cycle paths (Gouvernement du Québec 2021). One major 
challenge for the project has been maintaining traffic while construct- 
ing the new infrastructure and demolishing the old one. At the time 
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of writing this chapter, the project is almost completed, at a total cost 
of 3.67 billion CAD. The project included two phases. The first one, 
identified as a preparation phase and representing approximatively 20% 
of the overall budget, aimed to fix the underground infrastructure 
to facilitate the core work of the next phase. This phase was initially 
planned to run from 2010 to 2015, but was finalised only in 2019 (due 
to the problem with the sewage collector), with major impacts on the 
schedule and on the overall realisation strategy. Indeed, the second 
phase, which includes the main work on the interchange, was initially 
planned to follow sequentially the first one. 

The situation of interest to illustrate the knowing versus exploring 
trade-off concerns the condition of the major wastewater collector. 
The collector is part of the city wastewater network, and is situated at 
30 metres under an overpass which has to be demolished and rebuilt as 
part of phase 1. (Gouvernement du Québec 2015). In the design phase 
of the project, information from the city stated a fair condition for this 
collector. The only way to obtain accurate and reliable information 
would have been to get into the collector, which was not possible be- 
fore starting the physical work. Moreover, there was no reason to doubt 
the condition of the collector. So, in the development phase, the search 
for new knowledge was held back by a seemingly certain representation 
of the project based on information from the city, and by the absence 
of collector deterioration signals (weak or otherwise). So, design and 
planning continued without seeking additional knowledge. The second 
phase of the project was planned to start immediately after ending the 
first phase. Hence any delay in the first phase translated automatically 
into a delay in the second phase. To add to the complexity, the types of 
contracts were quite different between the two phases, to address dif- 
ferent markets. In the first phase, multiple conventional contracts were 
given to several local contractors following the usual bidding process. 
The specific contract for the overpass was attributed within the first 
phase at the end of 2013, for a start in April 2014. Conversely, in the 
second phase, only large consortiums submitted proposals for a contract 
of 1.6 billion CAD, attributed in December 2014. 

Only in 2013, when preparation works physically began on the site, 
giving access to the collector, did the problem come to light. A signifi- 
cant crack was observed on the collector. The ministry decided against 
starting the demolition of the overpass and causing more damage to 
the collector. Different options were analysed to either repair or de- 
viate the collector. By September 2014, the decision had not yet been 
made, and with the winter coming, repairing or deviating would have 
been impossible, leading to an additional delay of almost a year (Bisson 
2014). This had impacts on the local population, as the works were an- 
nounced and reported three times. Finally in 2015, the ministry made 


78 Monique Aubry and Serghei Floricel 


the decision to deviate the collector for about 1 kilometre. This re- 
quired complex works, including the use of a tunnel boring machine 
(Gouvernement du Québec 2015). The work on deviation effectively 
began in March 2015, and only in summer 2016 could the demolition 
and reconstruction of the overpass be started, two years later than in- 
itially planned (Bisson 2015; Gouvernement du Québec 2016). The 
additional 60 million CAD for the deviation of the collector included 
10 million CAD in contractor compensation for delays. 

In this case, for experts at the Quebec Ministry of Transport, the 
representation of this overpass had been clear enough with the avail- 
able knowledge, and they had not seen a need for more exploration. 
Further exploration on the collector would have taken more time and 
added costs to the development phase, because of the difficult access to 
the collector. Moreover, the representation of this overpass included a 
strong artistic aspect: the new infrastructure was expected to be ‘em- 
blematic’ of Montreal, with a cable-stayed bridge and high central 
mast, illuminated and recognisable for miles around (Bisson 2014). So, 
the attention of the experts might have been too focused on the visible 
part of the infrastructure which is attractive to the population, rather 
than on the problematic invisible infrastructure underground. 

In this vignette, what is of interest to the knowing versus exploring 
trade-off is that exploring an invisible infrastructure such as the sew- 
age collector was not needed to form the project representation: the 
knowledge provided by the city was judged as sufficient. Conversely, 
exploration took place on other elements of the project, such as the em- 
blematic cable-stayed bridge, which was strongly visible. Consequently, 
representation of such an aesthetic element took importance over the 
invisible collector. Overall, exploration was not performed at the same 
level and with the same intensity for all project components and at all 
stages of the project. Instead, there were variations depending, up to a 
certain point, on the strength of their representation. 


4.3.2 Pluralism versus support trade-off 


This trade-off refers to the compromise between allowing the broadest 
possible pluralistic front-end scrutiny to ensure that no aspect, interest and 
opinion is not taken into account, and maintaining the interest of core pro- 
ject supporters. A megaproject is a social and political endeavour as much 
as, or even more than a technical one. Undertaking the front-end is like 
crossing some sort of invisible boundary to enter a social arena in which 
project development and governance are submitted to close scrutiny. Best 
practice advises thoroughly investigating the needs of project users, and seek- 
ing feedback from the broadest possible range of stakeholders. Many such 
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recommendations have been included in regulatory frameworks, for example 
in the form of mandatory public hearings. But the social arena is a very dif- 
ferent world from that of technical and economic evaluation, in which mul- 
tiple stakeholders with pluralistic perspectives and competing interests create 
their own kind of uncertainty and complexity (Denis et al. 2007). While 
this scrutiny may prevent the emergence of acceptability issues in the later 
stages, the flurry of queries, suspicions, doubts and requests may be so intense 
that sponsors may have to bid farewell to their early visions. A new project 
representation will emerge and will be redefined as the front-end progresses 
(Cha et al. 2018; Morris 2013; Williams 2005). Moreover, several studies 
suggest that opening the project definition to a broader range of influences 
reduces its execution performance (Floricel et al. 2011) and results in a larger 
number of benefit-sharing and compensation claims (Gil & Pinto 2018). 

The changing project concept and diminishing gain perspectives may turn 
away some early supporters. Similar concerns stem from the logics of exter- 
nalising project representations. Best practice advises using various supports, 
such as material mock-ups and increasingly online, virtual and enhanced 
reality platforms to consult and even co-define the project with users and 
stakeholders. But once they are put out in the social arena, project representa- 
tions become symbolic artefacts (Floricel & Brunet 2019). They acquire a life 
of their own, difficult to control by their creators, as concrete embodiments 
of stakeholder expectations and implicit covenants with project sponsors. 
A variety of interpretations accrue to them, which stakeholders manipulate 
to advance their own interests, using mass media to amplify their social and 
political impact. This has a direct impact on front-end dynamics. In addition 
to assessing, the earlier the better, whether a solution is socially acceptable, 
developing and releasing project representations signals commitment to a 
specific solution and helps rally a coalition of stakeholders that support this 
solution. But the same representations also provide a target for competing 
interests and ideologically inspired opposition. Therefore, sponsors may hes- 
itate to unveil them or may tie their publication to certain dates or windows 
in the political calendar. They may also hesitate to develop and release more 
detailed representations because earlier, more ambiguous ones may look at- 
tractive to a broader audience. Any refinement of the project representation, 
for example defining the exact position of a transportation infrastructure or 
of a high voltage line may show some stakeholders that the project will di- 
rectly touch their vital interests, and may trigger organised and even violent 
Opposition. 

This trade-off suggests explanations for both paradoxes addressed in this 
chapter. Presenting too many alternatives and evoking the possibility of mul- 
tiple other opportunities may prevent the rallying of supporters around one 
option and may offer arguments for those opposing the project. Proactively 
developing detailed representations or presenting them at an inappropriate 
time is also likely to have a negative impact compared to a strategy of main- 
taining a degree of ambiguity until implementation begins. 
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Vignette 3: The memory of a material representation 


This vignette builds on the Turcot Interchange project. The situation 
concerns an early mock-up representation for the interchange that fea- 
tured a particular element. This representation was submitted to public 
scrutiny and was favourably received. But the element disappeared in 
the later representations because a change in concept rendered it useless. 
However, public expectations regarding the project continued to in- 
clude this element, and so the presentation of the new version, without 
the element triggered an uproar. 

The Turcot Interchange project was an example of the will to in- 
form and listen to local citizens and communities. They put in place 
a communication website where up-to-date information was posted. 
They also organised regular meetings with citizens to answer their 
questions, and accepted being challenged at times. The existing Tur- 
cot Interchange was an ugly (but unavoidable) infrastructure for road 
transportation only. The intention behind the new project was not 
to replicate the actual infrastructure — just the contrary. The global 
approach was to adopt a modern view on this sort of unavoidable in- 
frastructure in an ecosystemic understanding of its environment and 
sustainability. The project was seen as an opportunity to offer citizens 
a project to celebrate ‘Montreal colors’ (Gouvernement du Québec 
2010), including public transportation, cyclists and pedestrians. In 
line with this approach, the ‘Dalle-Parc’ was included in the project. 
Dalle-Parc is a large overpass covered by vegetation for pedestrians and 
cyclists. Moreover, it creates a link between different neighbourhoods 
which are otherwise difficult to connect, and gives access to a green 
space and landscape. The Dalle-Parc represents a budget of 40 million 
CAD, approximatively 1% of the total budget for the Turcot Inter- 
change project (CREMTL 2018). 

Back in June 2009, a mock-up of the whole Turcot Interchange, in- 
cluding the Dalle-Parc was presented to the public in the Turcot neigh- 
bourhood and was made available for visiting (MTQ 2009) as well as 
accessing a numeric version included in the project plan in 2010. In 
terms of representation, the visualisation contributed to making the 
Dalle-Parc very realistic, although some referred to these representa- 
tions as illusion! (Baillargeon 2010). 

Afterwards, a long silence followed on the Dalle-Parc, and around 
2015, some alarms were sounded when it was realised that it had sud- 
denly disappeared from the plan (e.g. Champagne 2015). Questions 
were addressed to the ministry to understand why such a decision had 
been taken. Interestingly, meanwhile, organisational changes had taken 
place within the project team and at the ministry. For example, a new 
project director arrived in 2014. With all these changes, it was difficult 
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to trace back the rationale behind the decision-making on the Dalle- 
Parc at the ministry. A formal explanation reported that costs were the 
primary reason. Mobilisation of the population became strong enough 
to put pressure on the ministry: 6,562 signatures were obtained on a 
petition to get the Dalle-Parc back into the Turcot Interchange project 
(Le Devoir 2017). In June 2018, the ministry confirmed the reintro- 
duction of the Dalle-Parc. 

In this case, the trade-off between pluralism and support turned 
around a material representation, which became a unique social rep- 
resentation of what the Turcot Interchange project should be for the 
vast majority of stakeholders, such as citizens in neighbouring areas, 
cyclists, communities, environmentalists, Montreal City Hall and so 
on. The initial representation was left unchanged, while in parallel, the 
project evolved as another representation within the ministry. Support 
and engagement towards a solution without the Dalle-Parc became less 
possible. Hence the ministry agreed to reintroduce the Dalle-Parc (and 
to add a new budget for it) to gain support from a variety of stakehold- 
ers. From then on, a common representation of the project returned. 


Vignette 4: When the freedom of individual cars is 
put into question 


This vignette reports a situation in which, among other battles sur- 
rounding the representations for a Quebec City public transportation 
system, environmentalist groups opposed the publication of an un- 
favourable public hearings report, fearing that it would jeopardise a 
project that could contribute to reducing greenhouse gas emissions. 
The report was eventually published, rejecting the solution proposed 
initially and triggering a reduction in project scope. 

The cost of this structuring of the public transportation system in 
Quebec City is estimated at 3.3 billion CAD. Its development phase has 
given rise to multiple and divergent voices. The core of the transporta- 
tion project is composed of streetcar (tramway) lines. Contrary to the 
first vignette, it was not a material representation of the streetcar which 
started and fuelled the controversy; it was, rather, the representation of 
the freedom associated with the use of individual car transportation: 
one might argue that Quebec City is a paradise for the individual car. 
The old part of the city is beautiful, and rich in patrimonial XVII Cen- 
tury buildings. Quebec City is also where the provincial government 
has its official offices. The population of the city is nearly 600,000 in- 
habitants compared with approximately 200,000 in the suburbs which 
have developed around the city (Ville de Québec 2021). Back in the 
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1950s and over time, several large avenues and highways were built to 
facilitate traffic circulation between the centre and suburbs. As in any 
cities, greater access resulted in more cars, nurturing a well-known 
vicious circle. In this context, it is not surprising that public transporta- 
tion is rather limited, with poor bus services and a lack of connections 
between the different circuits (Ville de Québec 2021). Moreover, it 
causes tension between downtown residents, who push for more public 
transportation, and suburban residents, who push for more investment 
in road access. In other words, a conflict exists between environmen- 
talists and proponents of individual cars. 

With the economic and demographic growth (pre-pandemic), the 
situation had become unsustainable. Around, 75% of daily motorised 
trips at peak hours are done by car (Ville de Québec 2021). The city 
cannot build more and more roads downtown and break the historic 
profile of Quebec City. In this context, in 2010, the Mayor of Quebec 
initially proposed the idea of a public transportation project identi- 
fied as the ‘Public Transportation Structuring Network’, in which the 
main component was a streetcar, well integrated within the overall 
transportation system in Quebec City. Since then, the project concept 
has wavered continually between including, or not, a streetcar, serv- 
ing, or not, the suburbs, and so on. These fluctuations were dubbed 
a saga by Martin and Moala (2021). On top of this, a change of the 
party in power at the provincial government level stopped the support 
the project had enjoyed from the previous party in government. The 
culminating point (as of now) was the publication of the unfavourable 
public hearings report in 2020. The hearings followed an anti-streetcar 
manifestation (Béland 2020). In short, the prime ministry imposed its 
own view on the project, including a streetcar serving the suburbs. 

This vignette illustrates the difficulty of managing the pluralism of 
perspectives on a project such as this one, while maintaining the sup- 
port and engagement of crucial stakeholders. Overall, the project is still 
going on, but has had to face several oscillations and changes. Here, 
multiple oppositions exist among different groups. The most important 
challenge concerns the suburban residents who are resistant to aban- 
doning their individual vehicles, and in so doing, are in opposition to 
the residents of the central city. With the change of party at a regional 
level, support for the project declined to a point that the prime minister 
took over the decision-making from the mayor of Quebec City. At the 
moment of writing this chapter, the story is probably not finished. As 
suggested in Paradox 2, there are consequent efforts and resources put 
in the front-end; knowledge is produced in a sort of back-and-forth 
process when solutions change and evolve. In doing so, the opportunity 
space is explored as different sets of stakeholders manifest themselves, 
describing a rather non-linear process. 
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This trade-off refers to the compromise between the efforts dedicated to 
produce a diligent, detailed evaluation of project solutions and perspectives, 
and the efforts committed to leveraging a particular solution in order to shape 
the future. Efforts dedicated to evaluation assume that the world has a given 
structure, and that additional knowledge production efforts can produce a 
more precise estimate of the benefits various solutions are likely to achieve 
in this world, and therefore would enable the selection of the best solution. 
On the other hand, shaping strategies assumes a fluid world, in which project 
outcomes depend to a large degree on sponsors’ efforts to structure the con- 
text in ways that favour any solution that happens to be selected. Prescriptive 
models and practices are mostly on the side of evaluation diligence, particu- 
larly in public projects where it expresses fiscal responsibility. But even with 
producing the most thorough knowledge and the most detailed and impartial 
representations of the project and its environment, benefits and cost are still 
uncertain (Flyvbjerg et al. 2004). 

As mentioned above, uncertainty may persist until digging and construc- 
tion is well under way, because drilling and tests give only partial informa- 
tion on soil conditions and material properties, while mock-ups or digital 
representations do not provide stakeholders with sufficiently realistic grounds 
to assess the usability and usefulness of the project. More importantly, com- 
plexity and the unexpected problems and events it causes are inevitable in 
major projects (Floricel et al. 2016). So, projects are bound to confront the 
unknown and the unexpected (Loch et al. 2006; Piperca & Floricel 2012), 
and even the strongest knowledge production effort and most careful ex ante 
exploration, representation and comparison of alternatives cannot guaran- 
tee that the best option has been selected. So, instead of emphasising end- 
less evaluation, which evokes a passive attitude that takes the surrounding 
world as given, project sponsors focus on shaping the world around them and 
making the project a compulsory passage point towards this future (Callon 
1986). Instead of reflecting a given future, project representations become a 
future-making tool, involved in practices of “imagining, testing, stabilising 
and reifying, through which abstract imaginings of the future are turned into 
a realisable course of action” (Comi & Whyte 2018, p. 1955). Practices such 
as framing, rhetorically manipulating and promoting project representations 
transform these into roadmaps that induce and prepare people to change their 
life-paths and habits in ways that ensure the success of the future. 

This trade-off also helps to account for both paradoxes addressed in this 
chapter. The relatively low investment in front-end activities suggests that 
sponsors may realise that endless debates over alternatives and evaluative re- 
finements do not guarantee better choices and projects, but focus instead on 
preparing the conditions for successful implementation and stronger impact 
for the selected solution. As evidence for this trend, we can mention the 
idea of projects meant to have a structuring effect on the urban tissue, and 
the call to include much broader indirect economic and social effects in the 
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evaluation of their benefits (Prager 2019), which means that almost any pro- 
ject can be justified. This trade-off also elucidates the paradox of the limited 
exploration of opportunities. From a shaping perspective, a rapid selection 
and even lock-in on a solution, instead of being undesirable because the solu- 
tion is possibly suboptimal, may in fact be desirable because it facilitates the 
structuring effect on the urban tissue and living patterns (Levin et al. 2012). 
Two vignettes will illustrate this trade-off. 


Vignette 5: A law to introduce an innovative 
governance framework and to avoid the 
usual governance practices 


We described above in Vignette 1 the rapid convergence on a path that 
largely reused existing infrastructure for the already mentioned REM 
in Montreal. This rapid convergence was only made possible by a series 
of efforts to impose this solution and remove any possible obstacles to 
its rapid execution. These efforts included a law giving special powers 
to project sponsors and the creation of special organisational structures 
to ensure the probity of contractors in the selection process and that a 
sufficient number of bidders would participate in the process. 

Since 2008, and updated in 2010 and 2014, the Government of 
Quebec has adopted a governance framework for its major public in- 
frastructures (Samset et al. 2016). The management of major infrastruc- 
ture projects (over 50 million CAD) falls under the Law for Public 
Infrastructure (Gouvernement du Québec 2013) and must follow the 
requirements at each step of the framework as documented in the di- 
rective (Gouvernement du Québec 2016). However, the Law for Public 
Infrastructure leaves room for exceptions made by the government. 
This is what happened in the case of the REM. 

The REM project opened for Quebec a new approach to financing 
and governing major public infrastructure investments. A new com- 
pany was created by law (Gouvernement du Québec 2017a) as a subsid- 
iary of the Caisse de Dépôt et Placement du Québec (CDPQ) which is 
responsible for the management and growth of the universal retirement 
Quebec pension plan (CDPQ 2021). The new subsidiary company 
name was CDPQ Infra. With this project, private (or quasi-private) 
money serves to develop, construct and operate the new infrastructure, 
not exactly as a PPP, but in an innovative governance approach, as the 
ownership of the infrastructure is kept under CDPQ Infra (CDPQ In- 
fra 2021). In the commercial agreement with the Government of Que- 
bec and l’Autorité Régionale de Transport Métropolitain (ARTM), 
CDPQ Infra has the monopoly on public transportation by tramway or 
light rail in the metropolitan area for 99 years (ARTM 2018). 
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With the special law, CDPQ Infra is allowed to manage the REM 
project outside the governance framework for major public infra- 
structure projects. The usual rules for bidders do not apply. Instead, 
specific governance structures and directives to protect confidential 
information have been put in place at CDPQ Infra for the selection 
process (CDPQ Infra 2018). These include the creation of an independ- 
ent committee on the probity and integrity of the process, evaluation 
committees made up of experts, and a selection committee made up of 
three experts to select bidders, based on the recommendations of the 
evaluation committees. 

As illustrated above in the first vignette, for the design team at 
CDPQ Infra, maximising the use of existing infrastructures dramat- 
ically accelerated the whole project despite criticism on its negative 
impacts. In this trade-off, we want to highlight specifically how a se- 
ries of decisions and practices have imposed this solution, and at the 
same time, removed obstacles to rapid execution. The first aspect re- 
fers to the governance structure adopted by law, with the creation of 
the CDPQ Infra given complete independence in their approach to 
the management of the project. With the exception of environmental 
regulation (BAPE), there is no compulsory governance framework to 
follow. Moreover, in the commercial agreement, the Government of 
Quebec and the ARTM granted to CDPQ Infra a monopoly on col- 
lective transportation via tramway or light railway in the metropolitan 
area. This latter agreement removes any further discussion on reusing 
the existing infrastructure. 

A second aspect of this project, aiming to accelerate the convergence 
towards solutions, is the contractual process. Current procedures for 
public contracts impose a strict procedure in order to minimise con- 
tractor fraud and collusion. This strict approach is recognised as being 
lengthy and bureaucratic, not encouraging a climate of confidence, and 
blocking contractors’ attempts to innovate. The special law, 2017, c17 
(Gouvernement du Québec 2017a) allows CDPQ Infra to use different 
procurement and contractual arrangements. The full bidding process 
required the consortiums to obtain a qualification before submitting 
their proposal. The project scope was split into two different calls for 
proposals: (1) EPC: infrastructure, procurement and construction, and 
(2) RSSOM: provision of rolling stock, systems, and operation and 
maintenance services (CDPQ Infra 2017). After the launch of the call 
for proposals to qualified consortiums, several delays occurred. The 
first delay occurred when the period for submitting a proposal was 
extended in order to give a chance for all qualified consortiums to sub- 
mit. Five proposals were received, two on the former call for tenders, 
and three on the latter. When proposals received for the EPC package 
tender were analysed, CDPQ Infra was dissatisfied because costs were 
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significantly higher than expected. So, a second delay occurred when 
CDPQ Infra decided to engage in intensive discussions with the two 
consortiums to work out a better quality/price equilibrium. In fact, the 
decision was announced publicly a few weeks later. In total, it took al- 
most two years between the announcement of this project in 2016 and 
the start of construction in April 2018. 

In conclusion, from this vignette, one can observe the success in 
accelerating project delivery by making a rapid decision towards the 
solution of reuse of the existing infrastructure. Other options were 
identified, but not studied in detail. Instead of putting effort into the 
evaluation of options at the development phase (and developing knowl- 
edge from these evaluations), the rapid decision was able to shape the 
project by leveraging the use of existing infrastructure and interacting 
with contractors (developing knowledge from this shaping process). 
This vignette highlights specifically the role of politics and govern- 
ance organisation in the shaping process. Without such mechanisms, 
it is doubtful that a swift convergence toward a single solution can be 
reached. 


Vignette 6: A lack of competency in estimation 
leads to hesitation 


The second vignette illustrates the case of a major tunnel refurbishment 
project in Montreal, in which only two of the pre-qualified consortia 
submitted offers. Of these, one withdrew during the evaluation process 
and the second made an offer that was significantly above the available 
budget. The first reaction was to postpone the project and organise a 
second call for bids. However, the Covid context and a report by the 
Auditor General of Quebec, criticising the persistent inability of the 
responsible ministry to develop and evaluate projects rapidly led to a 
reversal of this decision and the award of a contract for a newly negoti- 
ated price to the remaining consortium. 

The case illustrated in this vignette offers an opposite pattern to the 
previous vignette: the Ministry of Transport stopped the evaluation 
process (in the bidding process) based on the fact that there was no 
competition in the market. The sponsor was ready to pursue the evalu- 
ation process in a new bidding cycle. 

This project concerned the refurbishment of a tunnel under the St 
Lawrence River, the Louis-Hippolyte-La Fontaine Tunnel (LHLT). 
Montreal is an island, with several bridges to link it to the north and 
south shores, but LHLT is the only tunnel in Montreal, and in Quebec. 
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Consequently, local expertise in tunnels is rather sparse. The tunnel 
was built in 1967 for the Montreal Universal Exposition. It is a ma- 
jor road link not only for Montreal but for the whole of Quebec and 
Canada, for people and goods transiting to the United States. The full 
infrastructure includes an underwater tunnel, 1.5 kilometres long and 
a bridge 457 metres long. 

Since 2009, normal maintenance works in and around the tunnel, as 
well as water infrastructure and fire resistance tests signalled the need 
to undertake serious refurbishment of the infrastructure. The project 
scope was discussed with a wide variety of stakeholders, producing two 
main debates: (1) postponing this project for better coordination with 
other major infrastructure projects going on in Montreal to minimise 
the consequences in terms of deviations and travel time, and (2) moving 
away from a narrow view of this project of refurbishment of the same 
infrastructure towards a long-term perspective on sustainable mobility, 
including public transportation and active mobility. Finally, the opti- 
mal solution, and a design-build-finance execution mode was approved 
and presented by the ministry in June 2017 (Gouvernement du Québec 
2017b). A request for interest was launched in June 2018, from which 
two qualified consortiums were retained. They both confirmed their 
participation in the call for proposals. Two technical propositions were 
received in due time, in February 2020. However, only one consor- 
tium submitted a financial proposition by March 2020 (Le Fort 2020), 
the other having withdrawn from the bidding process. 

This was a turning point in the selection process. A single propo- 
sition was to be examined, but its price was largely over the planned 
ministry budget. This situation coincided with the onset of the pan- 
demic, which created fear and uncertainty. The option of delaying the 
project was now in the air (Bovet 2020). Moreover, the report of the 
Auditor General came out in June 2020. This report scrutinised 
the contractual process at the Ministry of Transportation and found a 
serious lack of expertise in cost estimation (VG 2020). The market con- 
text, characterised by recruitment difficulties, amplified the disconnect 
between the cost estimate that served as a basis for the call for proposals 
and the reality of the market. 

Following this report, the ministry decided to continue with the pro- 
ject of refurbishment. Two additional rounds of negotiations occurred 
with the sole consortium remaining in the process, before reaching 
agreement on a cost of 1.1 billion CAD. In June 2020, the consortium 
was Officially selected (MTQ 2020), with virtual meetings organised 
for the signing of the contract in July 2020 (Le Fort 2020; MTQ 2020). 

With regard to the selecting versus shaping trade-off, this case made 
a point of ending the evaluation by avoiding entering into a second 
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biding cycle; conversely, negotiations with the consortium leveraged 
the solution in order to shape the project. The ministry would have eas- 
ily entered into this new cycle, for reasons of consortium withdrawal, 
costs in excess of the budget, and uncertainty due to the pandemic. 
However, the Auditor General report and the precarious condition of 
the tunnel favoured the signature of the contract. 


4.3.4 Allocation versus collaboration trade-off 


This trade-off refers to the compromise between the efforts to develop a rep- 
resentation able to unambiguously allocate responsibilities and risks between 
participants and the efforts to create conditions for a collaborative elaboration 
and implementation of the project from its earlier stages. From an alloca- 
tion perspective, inspired by the prescriptions of agency theory, front-end 
practices elaborating detailed project representations are seen as the key to 
an effective transfer of responsibilities from sponsors or clients to contrac- 
tors. Project representations become boundary objects (Carlile 2002; Chang 
et al. 2013) which delimit the respective responsibilities and risks of project 
participants and allow contractors to evaluate whether they should submit a 
bid and for what price. In the case of design-build and related forms, these 
representations, somewhat less detailed, become the main interface between 
owners and execution (and sometimes operation) consortiums, while in tra- 
ditional design-bid-build mode, they are also a tool for allocating responsibil- 
ities and risks between contractors. Prescriptive models suggest that putting 
more effort into elaborating project representations helps clients, particularly 
public authorities, to limit their exposure to project cost increases and over- 
runs, and enables selection based on the lowest price and reliance on fixed 
price contracts. 

However, these practices have two shortcomings. The first is that the pro- 
cess of elaborating project representations is cut in two. The client, who 
brings an owner-operator perspective, aided by engineering experts, devel- 
ops the representation up to the call for bids, and then contractors, who have 
manufacturing, construction and assembly competencies, take over from 
that point, with relatively little change and collaboration possible afterwards. 
However, a key finding of front-end studies is that, especially for complex 
systems, projects have higher performance and fewer conflicts during im- 
plementation if solutions emerge from the collaboration of participants with 
different perspectives, such as owners, engineering firms and contractors 
(Hobbs & Andersen 2001). The second problem with allocation approaches 
is the fact that reliance on detailed representations and the network of fixed- 
price contracts based on them creates a rigid structure that favours adversarial 
relations and precludes adequate responses to unexpected events (Floricel & 
Miller 2001). The reaction to these problems is the emergence of governance 
frameworks and contractual forms that favour collaboration and risk sharing 
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(Mesa et al. 2016). These forms place less emphasis on the early development 
of detailed representations and their boundary object role, as it is expected 
that representations will be elaborated collaboratively and that contracting 
will be essentially incomplete and relational (Davies et al. 2019). But putting 
together a group of partners with the necessary competencies, and connect- 
ing them by a multi-party contract, with joint responsibility for the entire 
project from beginning to end may require early selection and convergence 
on a conceptual solution. An early representation of the project solution is 
also needed in order to provide an epistemic object that will help mutual 
understanding and the integration of contributions from participants with 
different perspectives. 

This trade-off also provides insights into both paradoxes addressed in this 
chapter. It may be too early to assume that the relatively low investment 
in front-end activities is due to the new governance and contractual forms 
which integrate what were typically seen as separate front-end activities with 
activities typically depicted as implementation. However, we can assume that 
such low investment may be due to the fact that project sponsors understand 
that no representation and contract, however elaborate, will allocate risks 
perfectly. Instead, additional resources will be needed in order to solve prob- 
lems as they appear during implementation and react to other unexpected 
events. Likewise, the limited exploration of opportunities may be caused by 
the fact that, in the most common approaches, only one perspective — that of 
the owner-operator — dominates the exploration process. Moreover, in order 
to avoid excessive delays, the exploration of alternatives has to make way for 
the elaboration of a unique and rather complete representation that will serve 
as a basis for the contractual process. This trade-off will be illustrated by two 
further vignettes. 


Vignette 7: An innovative governance for 
collaboration 


We continue with the REM case. The REM adopted what CDPQ In- 
fra calls an “innovative governance” which differed from the traditional 
approach (design-bid-build) or the PPP. In their approach, CDPQ Infra 
is the owner and the operator of the infrastructure (CDPQ Infra 2021) 
along with its partners. Partners are essential, as CDPQ Infra does not 
have the necessary capabilities in the domain of transportation, but a 
strict allocation of responsibilities might not be feasible. The adoption 
of a collaborative approach was illustrated in the bidding process. 
When proposals were analysed in the case of the first tender for 
the EPC contract, CDPQ Infra was dissatisfied with the proposals re- 
ceived: costs were significantly higher than expected (Labbé & Robidas 
2017). Despite the ensuing delay, CDPQ Infra decided to engage in 
intensive discussions with the two consortiums to come up with a best 
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equilibrium of quality/price, and the decision was announced publicly 
a few weeks later. The project representation evolved with the con- 
tribution of one or other consortium. Efforts put into the refinement 
of the solution with the consortiums could not have happened in an 
allocation perspective, where each contractual side works with what 
they think to be an unambiguous solution. Finally, the quality-cost 
equilibrium is better achieved via collaboration. Regarding Paradox 4 
on exploring the opportunity space, this case illustrates how it can also 
happen in collaboration with partners around a specific solution. 


Vignette 8: The digital technologies associated with 
a collaborative design approach 


This vignette will discuss the attempt of the Society for Quebec In- 
frastructure (SQI), a government-owned organisation, to develop a 
front-end governance framework that would enable the collaborative 
development of major project solutions. 

The SQI is a state-owned organisation which has the mandate to 
support 120 public organisms in the management of their public in- 
frastructure projects, including planning, realisation and rigorous 
monitoring (SQI 2017). SQI assumes direct project management for 
all building infrastructures, but plays a quality assurance role in trans- 
portation projects. The SQI reports directly to the Treasury Council. 

Over the last few years, the SQI has introduced innovative practices 
in relation to the digital transformation of the construction industry in 
Quebec. It is not a simple thing to transform an industry reputed to be 
a dinosaur in terms of productivity. Not only do digital technologies 
require specific competencies, but they also call for constant collabo- 
ration among stakeholders engaged in project design and construction. 
Collaboration is the key to success (Brunet & Forgues 2019). SQI has 
developed a Building Information Modelling (BIM) platform together 
with an integrated design process, and has engaged in a progressive im- 
plementation strategy by working closely with universities and industry 
(SQI 2021). In 2020 they were awarded the Best in Innovation Award 
over all categories at the Canada BIM Council (CanBIM 2020). 

With digital solutions such as BIM, together with an integrated de- 
sign process, the same representation of a project can be visualised and 
shared at the same time by several stakeholders in different spaces, as it 
evolves over time. This combined approach, along with a progressive 
implementation strategy, might ease the tensions created by allocation 
versus collaboration trade-off. Indeed, these practices reinforce invest- 
ments in the front-end in the form of collective sense-making between 
owner-operators and contractors. 
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4.4 Some reflections and lessons learned 


The trade-offs presented above were illustrated with vignettes describing 
current project situations. But what about new challenges anticipated in the 
future? In this section, we develop some reflections inspired from the above 
trade-offs to face the new (or simply different) paradoxical situations. 


4.4.1 Thinking the front-end in the context of sustainability 


The quest for a more sustainable environment should reinforce the front-end 
process. One can observe that many countries do adopt sustainability law for 
their investments in infrastructure. In developed countries, laws are adopted 
to compel projects sponsors to follow the United Nations 17 goals to trans- 
form the world (United Nations 2021). For developing countries, multilateral 
development banks require complete documentation to be provided on pro- 
ject sustainability before engaging their funds. Moreover, some scholars have 
undertaken research to explore how to include sustainability assessment in 
the overall performance of programmes and projects (Martinsuo & Hoverfält 
2018). The interest and will to engage in such a change seem to be there. 
However, consideration for sustainability calls for efforts and resources at the 
early stages of projects. Indeed, one of the main roles of the front-end is to 
make a clear assessment of why major investment is required at the proper 
strategic level. What is the problem to solve? In a sustainability approach, 
building no new infrastructure might be the ‘best’ option. It may involve 
‘deconstructing’ a representation. 

Thinking the front-end in the context of sustainability has a tight relation- 
ship with trade-off on knowing and exploring. Sustainability offers the op- 
portunity to explore not only more options but rich options with the inclusion 
of social, economic and technology aspects. 


4.4.2 Front-end governance to prepare for flexibility 


Even carefully worked out solutions encounter major obstacles, which need 
tactical efforts to deal with them. But obstacles agglomerate towards the end 
of the execution phase, as the project encounters ‘real life’. Rather than end- 
lessly refining the solution choice, which after such unexpected events may 
not look the best, front-end ‘governance’ has to put in place the conditions 
that allow responsive actions, which is rarely done in an explicit manner, and 
indicates the persistence of uncertainty. This point also includes the relational 
contractual arrangements. 

Front-end governance to prepare for flexibility relates to trade-off on 
evaluating and shaping, and the trade-off on allocation and collaboration in 
contractual aspects such as relational contractual arrangements (Davies et al. 
2016). 
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4.4.3 Innovative technologies 


The pluralism-support trade-off highlights the difficulty of engaging a vari- 
ety of users and stakeholders by visualising a conceptual solution. Innovative 
technologies such as Building Information Modelling (BIM) offer a shared 
platform among dispersed actors. However, an unexploited advantage of 
these technologies may be the possibility to enable collaboration with stake- 
holders at the very early stage of a project (Brunet & Forgues 2019). Inter- 
estingly, technologies are not at the core of the innovation in infrastructure 
projects: collaboration among the plurality of stakeholders remains the major 
component. A collaborative approach such as the Integrated Project Delivery 
goes in a similar direction (Walker & Rowlinson 2019). 

Yet the inclusion of innovative technologies in the front-end emphasises 
the relation with trade-off on pluralism and support. 


4.4.4 Competencies to deal with paradoxical situations 


In pluralist situations, there is rarely a clear path towards the ‘best’ conceptual 
solution. In major projects, actors in situations of leadership in the front-end 
activities constantly play around with ambiguity and uncertainty. This hu- 
man aspect is rarely addressed in the context of major projects. 

This orientation on competencies to deal with paradoxical situations 
crosses all four mentioned trade-offs, as it refers to a general attitude to ac- 
cept the two faces of reality and to advance the project in any way possible. 


4.5 Conclusion 


The aim of this chapter was to provide a sense of reality when undertaking 
the front-end in a major public project. The reality in the context of major 
public projects is rarely simple, given that these projects last for a long pe- 
riod of time and generally involve a plethora of stakeholders. So, the notion 
of paradox is excellent when talking about reality in this context: issues, 
tensions or conflicts are common in the course of front-end activities. This 
chapter addresses specifically Paradox 2, ‘the significance of front-end man- 
agement’, where the critical importance of the front-end is contrasted against 
its limited resources, and Paradox 4, ‘opportunity space’, where the selec- 
tion of conceptual solution is made without exploring the opportunity space 
(Samset & Volden 2016a). Yet major orientations and sensible decisions are at 
the core of the front-end activities. We rely on the notion of representation 
to anchor those paradoxical situations that we identified as trade-off. The 
structure of the chapter was built around four trade-offs where we followed 
the development of representations. Each trade-off was explained based on 
theoretical perspectives. Two vignettes, or short accounts, were provided for 
each trade-off. The vignettes were chosen for their potential to illustrate 
real situations occurring in front-end activities. Some reflections were then 
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offered to explore how those trade-offs might help when undertaking major 
public projects in the future. 

There are some limits in our approach. While it is a good strategy to offer a 
variety of complex situations, it was not possible to enter into their complex- 
ity in depth. We hope that the chapter has covered some caveats identified 
in a recent review of the literature (Williams et al. 2019). In this period of 
pandemic, governments are planning to invest massively in major projects as 
a strategy to restart their economies. This strategy goes hand-in-hand with 
strategies for the acceleration of project delivery. Care should be taken not to 
return to the time of bad projects (Miller & Lessard 2001; Morris 2013). With 
the introduction of national governance frameworks in several countries over 
the last decade or so, we have learned to better manage major public projects 
(Samset et al. 2016). Other lessons are still to be explored. 
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5 Estimation 
Richard J. Kirkham 


5.1 Introduction 


Of the ten ‘front-end’ paradoxes characterised by Samset and Volden (2016), 
this chapter is concerned primarily with ‘the cost estimation paradox’ — that 
is, the focus on the final cost estimate (the budget), at the expense of early- 
cost estimates. We consider other paradoxes too, including ‘the paradox of 
disregarded analyses of costs and benefits’ and the ubiquitous ‘success par- 
adox’ primarily through the lens of major public projects. A rich vein of 
academic scholarship and ‘grey-literature’ exists and provides a timely op- 
portunity to critique the well-established ‘lines of defence’ governance and 
oversight arrangements that surround the delivery of high-risk projects and 
programmes in the UK (Vo et al. 2021). 

The basis for the authorisation of funding public projects and programmes 
is a business case that demonstrates the benefits and disbenefits of the invest- 
ment relative to costs, using ‘social cost-benefit analysis’ (SCBA) techniques — 
the appraisal of ‘public value’, as it is also known. SCBA is based on the 
principles of ‘welfare economics’ (Hicks 1939) and is concerned with overall 
social welfare efficiency rather than merely market efficiency (HM Treasury 
2020). Public value therefore includes all substantial costs and benefits that 
affect the welfare of citizens as distinct from simple market consequences. 
The determination of costs and benefit elements according to the appraisal 
of social value requires varying degrees of estimation and forecasting which, 
by implication, introduce uncertainty, risk and complexity. The tendency for 
public projects to suffer from a predisposition to cost and time overruns has 
led to a rich vein of academic enquiry into causality, the literature pointing 
to a set of often deep-rooted and complex issues, including the genesis of the 
initial estimates of cost and duration against the agreed scope and strategic in- 
tent. We know that estimation takes place against a background of cost enve- 
lopes, contingencies (risk) and probabilistic methods, and as such, there will 
always be events which cannot be foreseen but which will nevertheless im- 
pact on a project and the long-term outcomes that it intends to deliver. The 
challenge of reconciling credible estimates with procurement routines which 
may favour the ‘race to the bottom’ remains an intractable problem, despite 
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academic scholarship and legislative reform (e.g. Social Value Act 2012, Fu- 
ture Generations Act (Wales) 2015) that has attempted to promote a holistic, 
‘whole life cost’ view of investment appraisal in major projects (Boussabaine & 
Kirkham 2004). 

Allied to this, the treatment of risk and uncertainty in the context of esti- 
mating is often simplified through largely reductionist thinking and methods. 
Makridakis et al. (2019a) suggest that the “greatest challenge is to evaluate 
all risks, which are of very different underlying natures, and determine their 
overall influence to propose practical actions, both operational and strategic, 
that can minimise their total negative impact and exploit the available op- 
portunities”. However, doing so requires one not only to recognise the lim- 
itations of attempting to predict the future but also to be able to determine 
the uncertainty of such events materialising, assessing their specific impacts 
and temporal interactions, and ultimately determining how they will interact 
in the organizational setting. Thus, managing risk is a highly complex and 
sophisticated endeavour that relies on forecasting accuracy, which, in turn, 
determines the extent of the future uncertainty, and eventually influences 
risk. In the context of government projects, we know that political influence 
and churn are contributory factors to systemic risk and uncertainty (Bloom- 
field et al. 2019), and that parliamentary cycles will affect policy decisions 
and fiscal commitments to projects and programmes, and yet our ability to 
counter their negative effects is usually highly constrained. 

We problematise the cost estimation paradox in the context of a well- 
established conundrum, that is, the naïve desire among some stakeholders 
for early (often unrealistic) commitments to time, cost and scope on the one 
hand, and the inevitability of low confidence in the achievement of those same 
metrics on the other. In this chapter, we explore the paradox through a series 
of ‘interrelated problems’, supplemented by six case studies — each intended 
to illustrate the challenges of cost estimating and forecasting in the context 
of the work of government across different project contexts and settings in 
the UK. These interrelated problems bear a similar resemblance to the find- 
ings of a 2016 study by Andersen, Samset and Welde — ‘Low estimates—high 
stakes: Underestimation of costs at the front-end of projects’, which con- 
cludes that “underestimation in the front-end phase was significant ... and 
poses a serious problem in that suboptimal projects are approved”. The study 
identifies underestimating risk, overestimating opportunities, inadequate es- 
timation methods and skills, reliance on weak information, and strategic/ 
deliberate scope creep and division of projects as antecedents to inaccurate 
cost and time estimates. Intriguingly, the results also evidence the problem 
of ‘pet projects’, these being characterised by unrealistic early estimates and 
‘disputable relevance’. 


5.2 The search for certainty in an uncertain world 


In his 1726 book, The Political History of the Devil, Daniel Defoe opined that 
“things as certain as death and taxes, can be more firmly believed” — a phrase 
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that is, perhaps, more commonly attributed to US ‘founding father’ and pol- 
ymath, Benjamin Franklin, who, in November 1789, wrote to the French 
scientist Jean-Baptiste Le Roy on the matter of the ratification of the Consti- 
tution and the start of a new government: 


Our new Constitution is now established, everything seems to promise 
it will be durable; but, in this world, nothing is certain except death and 
taxes. 


It is fair to say that human beings naturally find comfort in certainty, and 
some would argue that we are ‘hard-wired’ to seek it in an almost intuitive 
way. Thus, in the context of this book, an appreciation of the rich vein of lit- 
erature on the psychology of risk is relevant since it may provide insights into 
why. The Nobel Prize-winning behavioural economist Daniel Kahneman in 
his influential book Thinking, Fast and Slow offers many fascinating insights 
into decision-making under conditions of risk and uncertainty by focusing 
on the impact of biases and how these influence and shape human decisions 
and behaviour. 

Kahneman’s earlier collaborations with Amos Tversky (see Kahneman & 
Tyversky 1979) in the field of risk psychology highlight the many important 
considerations in framing our understanding of how people and organizations 
make decisions in uncertain conditions. Despite the prevalence of modern 
advances in computational simulation, forecasting and planning software, it 
remains the case that humans are ultimately responsible for making decisions 
on projects and programmes: “the control of risks associated with human fac- 
tors will affect project and portfolio success” (Merna & Al-Thani, 2005), and 
therefore we elaborate further the concept of ‘risk psychology’ in this section. 

Risk taking “is any consciously or non-consciously controlled behaviour 
with a perceived uncertainty about its outcome, and/or about its possible 
benefits or costs for the physical, economic or psycho-social well-being of 
oneself or others” (Trimpop 1994). Individuals exhibit unique perceptions 
of risk, but are generally categorised as being risk averse, risk neutral or risk 
seeking. Risk management is shaped by the decisions that humans make, and 
thus our understanding of the way people think is crucial. Numerous theories 
in psychology and economics have sought to facilitate our understanding of 
human perceptions towards risk, including the well-known theory of games 
and prospect theory. The onset of the SARS-CoV2 global pandemic em- 
phasised the importance of risk communication and how individual citizen 
behaviours are influenced by ‘framing’, the cognitive processes that establish 
problem boundaries in a decision-making situation. Tversky and Kahneman 
(1981) described how decision problems in which individuals systematically 
violate the requirements of consistency and coherence can be traced to the 
psychological principles that govern their perception of decision problems. 
An earlier study involving groups of students who were asked to estimate the 
speed of a vehicle in collision — but with a stimulus question that involved 
nuanced statements — revealed that stronger adjectives in the stimulus led to 
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higher average estimates of speed (Loftus & Palmer 1974). We also know that 
individuals tend to select inconsistent choices depending upon the framing of 
a question and in terms of concentration on losses or gains (Plous 1993). Ad- 
ditionally, bounded rationality is also an important concept to understand the 
complexity and lack of information which exist in the decision-making pro- 
cess. In general, traditional rationality demonstrates the assumption that the 
decision-maker understands the entire decision problem, including all possi- 
ble plans of action and their consequences. Moreover, the decision-maker is 
assumed to possess well defined preferences over final outcomes, and there- 
fore chooses optimally according to those preferences (Diasakos 2007; Gig- 
erenzer & Selten 2002). For all but the simplest problems, making optimal 
choices arising from rationality requires extraordinary cognitive and com- 
putational abilities and resources. However, as problems become simpler, or 
resources (costs) are used to identify more information about the problems, 
the decision-making choices are more closely derived from rational decisions. 
Unfortunately, since the decisions regarding all aspects of problems are made 
by humans whose rationality is limited by the information they have, it be- 
comes obvious that the decision-making process is likely sub-optimal. 

Furthermore, heuristics facilitates the resolution of complex decision prob- 
lems and is viewed as being close to the optimum. In psychology terms, 
heuristics enables individuals to make decisions regarding complex problems 
in situations of incompleteness (Gilovich, Griffin & Kahneman 2002). Kah- 
neman’s theory points to fundamental limitations in human mental processes, 
and thus causes us to employ various simplifying strategies to ease the burden 
of mentally processing the information required to make judgements and 
decisions. Thus, decision theory enables us to identify individuals’ percep- 
tions towards risk and uncertainty. For decisions under uncertainty, decision 
theory provides two explanations: an optimistic decision-maker considers 
the best possible outcome for each course of action and chooses the course of 
action that corresponds to the best possible outcomes, whereas a pessimistic 
decision-maker will consider the worst consequence of each possible course 
of action and select the one that has the least bad consequence. 

Moreover, the notion of regret is directly associated with decisions under 
uncertainty. Most people tend to regret deciding about something after real- 
ising the existence of something better. In economics, this is also known as 
the opportunity cost. Opportunity cost is “the cost of an activity measured 
in terms of the best alternative forgone” (Sloman 2004). Lastly, when the 
possible outcomes of a decision are given by probabilities, that decision is 
one taken under conditions of risk. This process requires calculating “the 
expected value associated with each possible cost of action” resulting in an 
action which gives the “highest expected value”. 

Kahneman’s recognition of these conditions and the inherent biases that 
affect our decision-making are elegantly captured in his concepts of ‘System 
T and ‘System 2’ thinking. This recognises that most humans do not rel- 
ish the prospect of pondering on complex, often intractable problems that 
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characterise the delivery of major projects; instead, decision-makers prefer 
to be ‘cognitive misers’ (Soll, Milkman & Payne 2015) by circumventing 
the mental energy required in evaluating uncertainties. The consequence 
of this is a simple one: decision-makers are constrained by a cognitive error 
that advocates for one possible future (i.e. the specific project that emerged 
from a policy announcement), one objective (building the business-case to 
fit the project), and one option in isolation (the pre-determined project). 
Decision-makers hence become victims of ‘tunnel-vision’ — influenced by 
the dominance of System 1. Intuition overrides any obvious necessity for 
more reasoned evaluation, and thus the project proceeds with a degree of 
baseless confidence. Cognitive alertness to bias in our decisions is crucial to 
expanding our views on all three fronts. Most of us tend to be overconfident 
in our estimates (optimism bias), and therefore it should not be surprising 
that over-optimism at the early stages of a project is a recognised antecedent 
to project ‘failure’. Therefore, we must ask ourselves why we observe such a 
deep-rooted desire to present highly uncertain variables such as cost and time 
as deterministic values and to place such high levels of confidence in their 
accuracy. It seems perverse to do so, knowing what we do — that the work 
of government is truly complex and that deterministic forecasts in business 
cases are generally inflexible to changes in the policy landscape upon which 
a major project is often predicated. 

Here, we suggest that there are two issues at play, cognitive biases and 
motivational biases coupled with the enduring problematisation of the com- 
munication of risk (Makridakis et al. 2019b), arising chiefly from the ambi- 
guities that surround the use of the term ‘risk’ and its disambiguation from 
other terms such as uncertainty, frequency, hazard, survival and probability. When 
coalescing these problems within a politically influenced landscape where 
projects are often promoted because of an electoral manifesto pledge or such 
like, one may appreciate how biases drive the obsession for cost and time 
certainty at the expense of realism and pragmatism. 


Case study 1: Over-optimism in UK Government 
projects 


In 2013, the NAO published a study into a long-standing problem in 
government, the initiating of projects based on unrealistic business- 
case assumptions. Notwithstanding a recognition of the problem and 
the treatment in the UK Treasury’s “Green Book’, the report criticised 
“frequent over-optimism”, which results in the “underestimation of 
the time, costs and risks to delivery and the overestimation of the ben- 
efits”. It undermines value for money at best, and in the worst case 
leads to unviable projects. The recognition of bias and cognitive error 
(whether unconscious or deliberate) is explicit throughout the report, 
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which argues that decision-makers should be intolerant of optimism 
biases in the delivery of projects, but also recognises the influence of 
“organizational behaviours and incentives and the strength of personal 
accountability”. Prior (to 2013) reports by the NAO illustrate a pre- 
disposition of failing to appreciate complexity in the front-end stage 
of projects, and, as a result, over-estimating delivery capability. More- 
over, there is a recognition that government commits to a ‘solution’ 
without fully understanding the context and exploring alternative op- 
tions. Similar observations were recorded in written evidence to the 
UK Parliament’s Public Administration and Constitutional Affairs 
Committee (PACAC) inquiry into the government’s management of 
its major projects (2020). Academics representing the Economic and 
Social Research Council (ESRC) funded Project X described a very 
similar problem: 


The ‘front end’ of any project is profoundly strategic. It is when 
we (should) define what is to be achieved, establish its feasibility, 
vulnerability (Bloomfield et al. 2019) and acceptability, and shape 
what is meant by ‘success’ (defined in terms of strategic perfor- 
mance rather than deliverables). Our review of the literature re- 
garding the “front end’ of projects (Williams et al. 2019a) evidences 
the persistence of predetermined solutions above rigorously eval- 
uated concepts that create public value. We identify a set of often 
neglected initiation (and completion) ‘best practices’ that should be 
in place before any major public project begins. 


The 2013 NAO report went further, emphasising that “half of [NAO] 
reports on major Government projects refer specifically to issues with 
estimating”, including the controversial High Speed 2 project to con- 
nect London with Birmingham, Manchester, Leeds and York. The 
report identifies five factors that contribute to over-optimism: inde- 
pendent challenge and accountability, complexity, stakeholders, evi- 
dence base and behaviours and incentives. The report recognises that 
tension often exists between the responsible use of public funds and the 
importance of innovation and risk-taking to achieve transformational 
change. It suggests that decision-makers must demonstrate confidence 
in the provenance of their decisions using ‘realistic’ estimates and as- 
sumptions. The report concludes with a warning that “optimism bias, 
and the desire to show success, may lead decision-makers to push on 
and deliver something even if the outcome is likely to be materially 
different from, and inferior to, that originally intended”, often known 
as the ‘sunk-cost fallacy’. 

The issue of realism is captured in the PACAC report, particularly 
in relation to ministerial decisions around policy and their implications 
for project delivery, one of the main concerns being the importance of 
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respecting democracy while practising the highest standards of govern- 
ance and transparency in public project delivery: 


Projects are hindered by over-optimistic estimates of cost and time 
schedules, and overstatement of early benefits. Ministers are too 
keen to commit to specific cost and timescales early in the process, 
and project managers become tied to these estimates. The early 
estimates can then shape the rest of the project delivery, sometimes 
leading to reductions in outputs or benefits as project-managers 
struggle to keep project timescales and costs in check. 


5.3 Reconciling the need for, and the paucity of, useful 
data at the front-end 


In the previous section, we argued that clear definition of the ‘front-end’ 
project or programme activities is crucial to creating the necessary conditions 
for a successful project and the delivery of benefits. It is also widely accepted 
that the opportunity to change the scope of a project (and thus influence cost 
and schedule) is at its greatest in the early stages of the project lifecycle. In this 
section, we explore the paradox of reliance on early estimates before much of 
the initial front-end work is complete. 

Assumptions are a necessity in the development of early-stage business cases 
for projects in government — yet determining final budgets or tendering at an 
early stage of the process before design, exploration or scoping are complete 
represents a preoccupation with price-driven metrics and, when it occurs, a 
fundamental lack of understanding of the nature of major projects and their 
true cost and value. Using these to inform tendering in the anticipation that 
these estimates are achievable is in part driven by the enduring inducements 
to pursue lowest price tendering — or an over-emphasis on capital cost over 
long-term benefits. In this section we look at the issue of estimating and the 
implications for project delivery in government. 

Cost estimation in this context is one of the main precursors to determin- 
ing the viability of a project or programme. According to Pinto, cost estima- 
tion processes “create a reasonable budget baseline for the project and identify 
project recourses (human and materials) as well, creating a time-phased 
budget for their involvement in the project” (Pinto 2007). Therefore, it is 
obvious that cost estimation and project budgeting are inextricably linked, 
both aiming to secure the completion of the project on time, on budget and 
at the appropriate quality. These vital aspects of the project, according to 
Gray and Larson, “are the lifeline for control since they serve as the standard 
of comparison of actual and plan throughout the life of the project” (Gray & 
Larson 2008). 

Primarily, cost estimation is taken into consideration before the tendering 
process for a project begins. During this early stage, organizations tend to 
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possess less accurate estimates because of incomplete information 
and availability of data. Thus, promoters should be more vigilant to the 
potential inaccuracy of cost and time estimates. As the project moves for- 
ward from the conceptual phase into the execution phase, information flows 
change, and more data becomes available. As a result, time and cost accu- 
racy of the estimation should improve significantly (Winch 2009). Projects 
are generally defined as “complex, non-routine, one-time efforts to create 
a product or service limited by time, budget and specifications” (Gray & 
Larson 2008), and therefore these characteristics emphasise the importance 
of considering the implications of unexpected events on initial estimates. 
Hence, corrections and amendments of the cost estimates should be under- 
taken based on a continuous process. 

During the project appraisal phase, the project organization is concerned 
with two fundamental decisions. The first decision relates to viability, and 
whether the project needs additional investment. If the project is viable, then 
the second decision is concerned with feasibility studies to identify the most 
successful option of those available. As the project options are established or 
eliminated, the range of uncertainty should reduce, even though at this stage 
the process still involves decisions to be made under a level of uncertainty that 
is often beyond the influence of the decision-maker (Smith 2003). 

On the other hand, during the execution stage of a project, the promoter 
may become more fully aware of the project’s expenditure since the infra- 
structure facility has been designed, constructed, and commissioned. Due to 
the adequate information about several important aspects of the project, the 
cost estimation is closely aligned to the true cost of the facility. Nevertheless, 
the construction phase is perceived as one of the riskiest phases in every pro- 
ject of any nature; therefore, cost estimations can be again slightly out of bal- 
ance. For example, poor performance of a contractor could lead to knock-on 
effects such as delays in the project schedule — the estimator should therefore 
consider how the potential for unforeseen events may be incorporated into 
the cost estimate. 

Finally, the operation phase is seen as the most straightforward stage of the 
three. The risky construction phase is after all over. Moreover, the project 
facility has been commissioned and 1s now operating and generating revenues 
(tolls paid to use the motorway, charges to use a new power plant, etc.). Ac- 
cording to Burke, the accuracy of the estimation improves significantly as the 
project moves through the different phases discussed earlier. At the end of the 
project the cost estimation becomes the actual cost, which tends to be higher 
than the predicted one (Burke 2006) The desiderata for accurate estimation 
of time and cost (Gray & Larson 2008) include (Figure 5.1): 


e Efficient and effective decision support 

e The antecedent to a project delivery schedule 

e Determination of whether the project is worth doing (strategic alignment) 
e Forecasting cash flow requirements 
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Figure 5.1 The basis of estimating. 


e Understanding how well the project is progressing 
+ Developing time-phased budgets and project baselines 


At this stage, it is important to state some issues that affect the ability to 
produce reliable and accurate project cost estimates. The nature of the prob- 
lem often originates from the ‘uniqueness’ of a project and the consequential 
implications for innovation. Pinto (2007) suggests that the most common 


reasons for variations in estimates include: 


Unrealistic early stage estimates: in the previous section we emphasised 
the problem of optimism bias, but there are other reasons why initial 
estimates can be lower than the actuality. As a project progresses through 
the business-case phases, new information can emerge (often related to 
policy), with implications for time and cost. There may be cognitive is- 
sues that come into play, and the way in which estimators interpret data 
can influence the outcome. 

Unexpected technical difficulties: although an estimate is established in 
part to identify and mitigate the technical risks of a project we know 
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that these can often underestimate the true cost. Later in this chapter 
we explore one example of an IT project that was ultimately flawed by 
technical errors. 

e Clarity of definition: when the project has been created without clear 
and reasonable scope, objectives and purpose in place, it is the norm that 
the cost estimation and project budgeting process will be insufficient as 
well. 

e Specification changes: the calculation of the cost estimates according 
to the specifications received from a client can be highly susceptible to 
variation — this is particularly problematic in situations where a client or 
project promoter is ‘naïve’ and therefore inattentive to the impact that 
scope and specification changes have on estimates. 

e External factors: are mostly concerned with political, economic, social 
and legislative events that are beyond the control and influence of the 
project. 


The terms of reference of the 2020 UK Parliament Public Administration 
and Constitutional Affairs Committee enquiry into major project delivery 
in government focused on the issues of estimating and forecasting in govern- 
ment, and sought answers to two questions, in particular: (1) How well does 
the government estimate cost, time and benefits at the start of projects? and 
(2) Are there barriers to doing this well, and what mechanisms could be used 
to ensure estimates are more accurate? 

Written evidence submitted by ESRC-funded Project X academics! em- 
phasises a desire for ‘single-point estimates’ of time, cost and forecasted ben- 
efits, despite the material fact that such estimates are likely to be incorrect. 
The nature of major projects is such that deterministic estimates cannot ac- 
count for the stochastic behaviour that we have observed in many strategi- 
cally important projects and programmes — particularly those that that span 
parliamentary reporting cycles. Anchoring bias can lead to perceptions of 
success/failure that lie at variance with the actuality of what is achievable, 
given the capabilities of the project participants and the available resources at 
the time of approval. In some cases, projects are authorised without adequate 
consideration of the data requirements. 


Case study 2: National Offender Management 
Information System project (C-NOMIS) 


In June 2004, a newly formed executive agency of the Home Of- 
fice, the National Offender Management Service (NOMS) initiated 
an IT programme that was designed to support the implementation of 
a new method of working, known as ‘end-to-end offender manage- 
ment’. Known as C-NOMIS, the system would replace separate leg- 
acy database systems in the prison service and probation service with 
one new, fully integrated system, the principal benefit being to enable 
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real-time data sharing between the prison service and probation of- 
ficers and other relevant criminal justice system agencies. However, 
the programme experienced numerous challenges — leading to a real- 
isation that the system ‘go-live’ estimated for January 2008 would not 
be achievable. It transpired that the original programme cost estimate 
rose by 195% from £234 million in 2005 to £690 million in July 2007. 
In the subsequent NAO investigation into the C-NOMIS project, the 
department admitted to a “gross underestimate of costs and that the 
original cost estimates were badly prepared”. 


Estimated Estimated Estimated Cost Increase 
Cost June Cost July Cost Increase 2005—2007 (%) 
2005 (£ 2007(£ (£ million) 

million) million) 


Application 51 206 155 304 
development 
System 128 243 115 90 
maintenance and 
support 
Infrastructure 14 38 24 171 
Implementation 12 22 10 83 
Project 13 34 21 162 
management 
Value added tax 0 99 99 — 
Risk (contingency) 0 32 32 — 
Other costs 16 16 0 0 
(sunk costs, 
management 
information 
systems etc.) 
Total 234 690 456 195 


In the first case study, we highlighted how early estimates shape 
project delivery, sometimes leading to reductions in outputs or bene- 
fits. In this case study, we observe a similar problem in that the NAO 
‘value for money’ assessment of the C-NOMIS project was described 
as ‘poor’, suggesting that many of the causes of delays and cost overrun 
were foreseeable but occurred due to a lack of project delivery expertise 
within the agency. More specifically, the agency was criticised for a 
failure to appreciate the nature of the operational context in which the 
programme was situated, inadequate oversight of the project (at Senior 
Responsible Owner (SRO) level) and poor supply-chain management, 
leading to a three-year delay in programme roll-out and reductions in 
scope and benefits. 


110 Richard J. Kirkham 


5.4 Government projects are truly complex yet 
estimating their cost is deterministically influenced 


Earlier in this chapter, we emphasised the propensity for government minis- 
ters to commit to deterministic estimates when announcing new projects as a 
vehicle for the delivery of a policy intent — despite history suggesting that the 
accuracy of such estimates is often woeful. One of the causes, often quoted 
by scholars, is that of strategic misrepresentation — this may manifest itself in 
assumptions best described as ‘underdeveloped’ and optimistic forecasts of fu- 
ture long-run benefits. While an unrealistically low initial cost estimate may 
increase the chance of a project being funded, future problems are “baked in’ 
and are often irreconcilable. 

The question of why costs, benefits and time forecasts of complex projects 
are systematically over-optimistic in the front-end — termed ‘normalization 
of deviation’ by Pinto and Slevin (2006) — has pre-occupied much of the 
project studies literature over the past few decades. This variation during 
the front-end is important for the concept of ‘lock-in’. Costs are considered 
during the formal decision to execute a project, but often the actual decision 
to execute precedes this, so decisions are made based on earlier, lower cost es- 
timates. This ‘escalating commitment of decision-makers’ can occur both at 
the decision-making level (before the decision) and during project execution. 
Public projects are particularly prone to wider influences such as political and 
legislative factors in the project front-end, and this can lead to entrenched 
problems as the project evolves. 

Benchmarking techniques are often utilised by decision-makers in the de- 
velopment of assumptions and early estimates of costs, but historical bench- 
marking requires careful analysis and clear ‘health-warnings when applied 
to projects and programmes of high complexity. Government projects usu- 
ally require the co-ordination of human and physical resources drawn from 
multiple organizations in both the public and private sectors, the majority of 
which will not be core to the central project delivery team function in gov- 
ernment. This recognition is crucial to recognising that the work of govern- 
ment is inextricably complex, and that the desire for reliable cost and schedule 
estimates must be tempered with the recognition that complexity, by its very 
nature, introduces high levels of risk and uncertainty at the ‘front-end’ of the 
project or programme. The effective communication of risk and uncertainty 
is therefore essential, as is the responsible use of probabilistic methods (mod- 
els) in a format that is understandable to decision-makers. 

We can understand complexity, to an extent, through an appreciation of 
the interdependencies that occur at any given time during the project life- 
cycle. This is not an easy task however, particularly in very long duration 
projects and programmes where changes in policy may interact with budget 
constraints. Complex systems are ‘unknowable’, and thus we must learn to 
live with the consequences of complexity in the context of major government 
programmes. 
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Defining the scope is crucial to a positive future project trajectory. Strate- 
gic intent is clearly important too, revisions to the HM Treasury Green Book 
in late 2020 seeking to strengthen business cases through improved appraisal 
of strategic intent. Here, we have problematised the conflicting demands 
in achieving scope clarity at the very early stages of the project lifecycle, 
highlighting the importance of unambiguous and clear objective setting and 
appropriate measurable objectives that are aligned with the policy goals of the 
government of the day. The effect of parliamentary cycles should be factored 
into conceptualisations of successful scope development, and an appreciation 
that the availability of funding from the Treasury will influence the capability 
of the project delivery team to deliver against the agreed scope. A study led by 
Cranfield University and funded by the Association for Project Management 
(APM) on ‘the practice of governance’ illustrates this problem in the con- 
text of ‘fixed goal’ and ‘moving goal’ projects and programmes (Bourne & 
Parr 2020). 

Recent reports published by the National Audit Office illustrate some of 
the points raised heretofore, including the nature of government major pro- 
jects and how this introduces challenges to accurately estimating the time and 
resources necessary to achieve delivery. The desire to use ‘reference classes’ as 
a means of supporting cost estimating is well versed in the project studies lit- 
erature, but there are few appropriate comparators to draw upon, particularly 
in defence capability and government transformation type projects. A further 
problem related to scope is the influence of other government priorities and 
how these are often conflated with existing projects and programmes. Where 
programmes and projects form part of a portfolio, externalities can negatively 
impact upon the reliability of the initial estimates. 


Case study 3: Prison estate reform 


Her Majesty’s Prison and Probation Service (HMPPS) is an executive 
agency of the Ministry of Justice (MoJ), which embarked on ‘The Prison 
Estate Transformation Programme’ (PETP) to improve outcomes for 
prisoners in key metrics such as decency, safety, education and health 
care. The programme ran for almost three years before it was super- 
seded in August 2019 by a government announcement committing to 
build a further 10,000 prison places, in addition to those expected to be 
built under the PETP. The NAO assessed the programme as ultimately 
undeliverable. Consequently, the prison estate is not currently meeting 
the needs of the prison service, and therefore the programme represents 
poor value for money. 

The failure of the PETP was partially attributed to an ‘over- 
ambitious’ scope, characterised by a programme timetable that was 
overly optimistic and susceptible to interdependent projects within the 
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programme that were operating on a tight timeline. HMPPS planned 
to build five new prisons by 2020. However, there was little contin- 
gency in delivery, and key factors including planning permission for the 
new sites were not adequately factored in. The House of Commons Jus- 
tice Committee report ‘Improving the Prison Estate’, which examined 
the causes of failure in the PETP, points to fundamental issues of strat- 
egy and funding. The report describes the Prison Service as “operating 
hand to mouth”, by reacting to immediate crises rather than developing 
a long-term strategy for the prison estate. The report recognises that 
the Ministry of Justice was subject to significant budget cuts in the pre- 
vious decade, and that most of its spending on the estate was ‘reactive’ 
and crisis-driven rather than strategic. The paucity of available funding 
from the Treasury also led to delays early in the programme. 

The Justice Committee report also emphasises the complexity of 
the programme within a broader landscape of policy reform to reduce 
re-offending in the UK. It identifies an absence of cross-government 
strategy, pointing to the importance of a systems thinking approach 
that recognises that the causes of crime cannot be tackled by imprison- 
ment alone, but by a broader approach to tackling inequality, unem- 
ployment, education, mental health care and social services provision. 
The report describes a ministry that “is still reeling from the long-term 
consequences of its unrealistic 2015 Spending Review settlement”, and 
“bears the financial and human cost of sustained underinvestment”. 
Interestingly, we see further evidence of optimism bias in that the re- 
port suggests that the ministry admitted that its 2015 Spending Review 
settlement was ‘over-optimistic and fundamentally unbalanced’. Hav- 
ing overestimated its expected income and the level of savings it could 
make in other areas such as the provision of ‘Legal Aid’, it exposed the 
PETP to insurmountable problems from the outset. 

Others are drawn in through the supply chain because of a special- 
ism, pulling together to build an infrastructure asset which faces unique 
challenges. Making accurate forecasts around cost and scheduling is 
therefore difficult in cases where technology readiness levels are low. 

Government is over-reliant on cost (and, to some extent, time) in de- 
termining project success, and frequently seeks to reduce contingency 
funds to ‘save’ money — thus limiting flexibility where new informa- 
tion or data suggests an alternative course of action should be consid- 
ered, or if economic conditions change. 

The quality of an estimate can only be judged during the execu- 
tion of a project or programme. To expect an initial forecast to stand 
the test of time, where wide-ranging and uncertain variables come 
into play, is simply not realistic. However, steps can be taken to better 
align expected and final outcomes by understanding that the evolution 
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of project design and re-forecasting improves as the project is better 
understood. 

Promoting better understanding and management of project delivery 
processes, and adopting more inclusive measurement metrics beyond 
just cost value will help to reveal the true value of a government project 
and improve the project delivery team’s ability to manage significant 
change or disruption as it occurs. 


Case study 4: Crossrail 


The Crossrail project is a railway construction scheme which aims 
to provide reliable, high-frequency suburban passenger train services 
across London, traversing from Reading in the west to Shenfield in the 
east, with two spurs to London Heathrow Airport and Abbey Wood, 
the latter being a new construction. Crossrail connects two major rail- 
way termini, London Paddington (for the Great Western Main Line) 
and London Liverpool Street (for the Great Eastern Main Line). The 
project was approved in 2007, with construction commencing in 2009 
on the central section of the route (mainly underground tunnelling 
work and the associated infrastructure) and connections to existing 
lines. In 2016, the Crossrail route was given the operational name the 
‘Elizabeth Line’ at a ceremony attended by HM Queen Elizabeth II at 
Bond Street Station. 

The idea of building a west-east railway line in London is not new; 
the project’s origins can be traced back to the Second World War. Fol- 
lowing attempts in the early 1990s to gain legislative approval and ac- 
cess rights, the present scheme was developed by the devolved transport 
body, Transport for London (TfL) and the UK Government Depart- 
ment for Transport (DfT), commencing in 2001. Crossrail required a 
‘Hybrid Bill’ (a mix of a public bill and private bill), which was submit- 
ted to the UK Parliament in 2005 and progressed through the various 
legislative procedures before receiving Royal Assent in 2008. To de- 
liver the project, a new ‘arms-length’ delivery organization was created 
by TfL and DfT as joint sponsors, with the involvement of the wider 
rail infrastructure asset owner, Network Rail. 

The scale and complexity of the scheme presented Crossrail Ltd with 
a significant challenge in estimating the schedule, and therefore the 
costs of the scheme. Aside from the civil engineering challenges associ- 
ated with tunnelling in central London, the nature of the work required 
Crossrail Ltd to award 36 contracts to private companies to deliver the 
main infrastructure (permanent way, stations and signalling) and the 
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electrical power systems required to operate the railway. The contrac- 
tual arrangements created many interfaces and dependencies between 
the work-package contractors, the consequence being £2.5 billion in 
increased costs during the period 2013-2018, as delays and changes to 
the programme occurred. 

Earlier in this chapter, we explored issues of ‘over-optimism’ and 
how the actuality can vary significantly from the initial estimates. The 
Public Accounts Committee (PAC) and National Audit Office (NAO) 
reports into ‘Delivering Crossrail’ emphasised this point elegantly: 


In our April 2019 report ..., we concluded that the Department, 
TfL and Crossrail Ltd’s fixation on a delivery deadline of Decem- 
ber 2018 led to warning signs that the programme was in trou- 
ble being missed or ignored. We found that the pressure on, and 
determination within, those delivering the programme to meet 
this deadline led to an overriding culture of over-optimism and 
that when Crossrail Ltd and the programme’s project representative 
identified failings at a much earlier stage, they were not addressed 
as they should have been. 


The propensity to commit to deterministic estimates of time and cost 
early in a programme’s life is, and remains, an enduring problem in 
public projects. Clearly, competing pressures come into play, and the 
case of Crossrail is no different. Railway projects are particularly sus- 
ceptible, due to the complicated nature of the rail industry in the UK 
and the time-horizon associated with, inter alia, timetabling of rail ser- 
vices. Nevertheless, the literature regularly problematises this in terms 
of the obvious need for a programme to sufficiently develop to a stage 
where more accurate information on what is needed to deliver it be- 
comes available. The NAO reports identify the tendency for delivery 
organizations to seek to incentivise targets that are subsequently rec- 
ognised as aggressive. The issues identified in Crossrail bear a similar 
resemblance to the UK’s new high-speed rail project (HS2), the first 
phase of which (London to Birmingham) was originally scheduled to 
open in 2026, but this was demonstrably too ambitious, given the leg- 
islative and planning hurdles that exist. There was also criticism of the 
‘reference-class’ basis that determined the initial estimates. Aside from 
changes that were required to adjust the programme budget during 
spending reviews, it was found that “subsequent cost estimates did not 
account for the level of uncertainty and risk in the programme, result- 
ing in significant cost increases once contractors developed the detailed 
design of the programme to inform a bottom-up costing”. Phase One 
is now estimated to complete between 2029 and 2033 (note the use of 
a range of values) on 140 miles of new permanent way. 
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You choose your leaders and place your trust 

As their lies wash you down and their promises rust 
You'll see kidney machines replaced by rockets and guns 
And the public wants what the public gets 

But I don’t get what this society wants 

I’m going underground, going underground ... 


The lyrics to the 1979 hit ‘Going Underground’ by The Jam were borne 
out of band member, Paul Weller’s angst in the face of a British Conserva- 
tive Government led by the newly elected Margaret Thatcher MP, whom he 
perceived to prioritise public spending on nuclear proliferation over govern- 
ment programmes that delivered broader public value. The words also betray 
Weller’s contempt for what he perceived to be implicit citizen apathy that 
manifested itself in a lack of resistance to such policies. 

In modern democracies such as the UK, decisions on investment in major 
projects are often influenced by electoral (rather than economic) cycles. Po- 
litical parties use their manifesto pledges to entice voters with attractive pol- 
icies, and if successful, form a government with a mandate to deliver on their 
promises — usually through projects and programmes. It is right and proper 
that the government of the day is empowered to determine the projects and 
programmes it deems necessary to achieve its policy objectives, but the corol- 
lary is an obligation on ministers to appreciate the implications of their policy 
announcements, especially where the required capabilities, competences and 
capacity of the civil service and its supply-chain may not be fully understood 
or appreciated. There are many examples in the UK to illustrate that this is 
an enduring problem — here we hypothesise that ‘the public gets what the 
public wants’ in the short term, but that in some cases, the public good may 
be damaged in the long-run. 

The public good, often referred to as ‘public value’ is explored by Lee, 
Oakley and Naylor (2011) who reflect on Moore’s (1995) critique of ‘new 
public management’ (NPM), an ideology that rejected the hereto traditional 
bureaucratic model of public administration. Their work is important in 
highlighting the conceptualisation of what may constitute ‘public value’, or 
“how the working practices of public servants might contribute to particu- 
lar sorts of benefits found only in public services”. The authors posit that 
value may be derived by the provision of a new public service (a local bus 
service partly subsided by the local authority, for example) or increased trust 
in a public service (or the institution that provides it). We discuss this in the 
‘Horizon’ case study later in this section. 

Leadership is crucial to ensuring that scope, time, cost and benefits fore- 
casts are realistic in the context of government transformation. Kortanta- 
mer (2019) proposed a framework for responding to project and programme 
complexity based on surveyed experiences of government transformation and 
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service delivery. The analysis adopts the theoretical lens of routines as it ena- 
bles the examination of both formal leadership activities and their improvisa- 
tions in responding to complexity. 

Williams et al. (2019b) presents a cross-national comparison of public sec- 
tor benefits management (BM) practices in Australia, Canada, the UK and 
the US, exploring “benefits management practices in action’, and considering 
to what extent ‘espoused’ or ‘mandated’ frameworks are practised and per- 
ceived by their users. Employing qualitative analysis, semi-structured inter- 
view data were analysed from 46 participants with experience in sponsoring, 
managing and/or reviewing government projects. The results expose con- 
siderable variation in the adoption and standardisation of BM frameworks 
from inter and intragovernmental perspectives. The research evidence places 
a strong focus on benefits identification, specifically at the outset (the busi- 
ness case stage, seeking project approval) and a deterioration in focus as the 
project or programme progresses through the authorisation (or assurance) 
approval gates towards close-out and operations. A further study of three UK 
projects illustrates the complexity and fluidity of how benefits are defined, 
particularly where many are social and non-financial, within a changing po- 
litical, social and technological landscape, and practical recommendations are 
made to achieve projects’ planned benefits. The results further emphasise the 
prominence of political interest, leadership buy-in, a benefits-driven culture, 
a transparent benefit reporting mechanism and an effective ‘hand-over’ to 
the business-as-usual operational phase in the implementation of ‘effective’ 
BM frameworks. 

The public ‘at large’ tend to focus their attention on the ‘costs’ rather than 
‘benefits’ of public spending — and thus the pressure to deliver ‘on time, 
on cost’ is, ostensibly a pressure that is exerted chiefly by the public (possi- 
bly through their elected representatives). Ministers are held to account by 
elected representatives who, in turn, are empowered to question government 
on its project delivery performance. This conundrum leads us to seek to 
problematise the project performance versus project success tension from a wider 
socio-economic perspective. 

The socio-economic impacts of infrastructure have been extensively re- 
searched, both in terms of economic development and social benefits. While 
the two are closely intertwined, there are also substantial studies exploring 
the two independently. Thus, we will further divide the literature into the 
following two subcategories: social impacts and economic impacts. 

Although there is no consensus on specific types of impacts, there are 
several commonalities found in the studies investigating social impacts. Zam- 
ojska and Préchniak (2017) describe the social economy as covering em- 
ployment, social services and social cohesion. This is a sentiment shared by 
Dobson et al. (2020), who label it as social value with the added aspects of ur- 
ban environment improvement, and the nurturing of specialist supply chains. 
The researchers emphasise how the fundamental purpose of infrastructure is 
to meet societal needs, and as such, the benefits of infrastructure should not 
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be limited to the provision of its basic functional utilities. Additionally, The 
Social Value Act (2012) makes it a legal obligation for public sector projects 
to demonstrate how they add social value. 

Bristow and Nellthorp (2000) describe how social impacts also encompass 
environmental and public welfare and demonstrate how infrastructure di- 
rectly improves social welfare. They illustrate how, for example, improving 
the quality of transportation infrastructure is linked to the reduction of road 
accidents and traffic disruption, which consequently saves travel time and 
costs. In this case costs are divided into direct costs: damage to vehicles, prop- 
erty, medical expenses, etc., and indirect costs: loss of production through 
the loss of skilled or unskilled workers — and hence a cost to the economy. 
The research corresponds to the study by Aschauer (1989), which argues that 
infrastructure enhances the very quality of life, with direct improvements to 
public safety, appearance of urban environments, and health and well-being. 
Agénor and Moreno-Dodson (2006) also demonstrate a clear link between 
infrastructure and education, as well as accessibility to health care. They de- 
scribe how efficient infrastructure acts as a direct channel, in which accessi- 
bility and quality of health care and education for individuals and society are 
improved tremendously. 

Finally, utilities such as water and energy can be a significant portion of a 
poor household’s income. Foster and Yepes (2006) describe this as another el- 
ement of social impact, wherein investment into maintaining and improving 
energy and water supply infrastructure can also alleviate the financial burden 
faced by poorer households. Straub and Terada-Hagiwara (2010) explain how 
between one third to a half of infrastructure services are used by households 
as final consumption, playing an important role in poverty reduction and im- 
proving welfare. They demonstrate how infrastructure for utilities provides 
essential services, the absence of which is a significant dimension of poverty. 
Additionally, improving existing infrastructure can reduce overall produc- 
tion costs and maintenance costs, resulting in reduced utility prices. 

The relationship between infrastructure and economic impacts is usually 
encapsulated as economic development or growth, and this has been achieved 
in several ways. The most common method is to directly relate economic 
development to growth in GDP per capita (Straub & Terada-Hagiwara 2010). 
Alternatively, Agénor and Moreno-Dodson (2006) explain that infrastruc- 
ture induces the direct productivity effect wherein the costs for input factors 
in production processes are lowered. They also argue that this is accompanied 
by the indirect effect where the overall productivity of the workers is im- 
proved. In fact, the idea that infrastructure lowers input costs while generat- 
ing an overall increase in production is shared across several studies (Aschauer 
1989; Calderón et al. 2014; Khanna & Sharma 2021; Lall et al. 2004; Mac- 
donald 2008; Montolio & Solé-Ollé 2009). Muvawala et al. (2020) find that 
initial high costs lead to a negative impact on economic growth, although 
this overturns in the long term showing significant economic growth. The 
authors argue that the negative short-run impact only demonstrates that 


118 Richard J. Kirkham 


public spending needs to be rebalanced in order to better prioritise social sec- 
tors. Fan et al. (2002) share similar findings, with overall long-term impacts 
being an increase in GDP per capita and thus positive economic growth. 
They also acknowledge that this is largely due to notable regional inequalities 
and accessibility. 

A study by Zolfaghari et al. (2020) evaluated the effects of social and phys- 
ical infrastructure on income inequality, and found that investment in water, 
education, health care, energy and digital infrastructure had the largest im- 
pact in successfully reducing income inequality. Due to the magnitude of the 
effects varying across different infrastructure investments, they concluded 
that an optimal balance must be found in allocating physical and social in- 
frastructures, as an unequal distribution of resources may have an opposing 
outcome. In relation to this, Leigh and Neill (2011) found that infrastructure 
investment on a national scale had eventual spill-over effects which contrib- 
uted to reducing unemployment at a local level. While the results of the study 
were promising, they were unable to determine the effect on the national 
economic activity. However, the authors emphasise that there was evidence 
to suggest that an indirect link between better national infrastructure boosts 
local economic activity due to lowered transportation costs for independent 
local business, consequently leading to a substantial increase in job creation. 

There is further research examining the relation between unemployment and 
infrastructure investment, all studies appearing to be of similar disposition — 
that efficient investment in infrastructure leads to an overall decrease in un- 
employment and increase in job creations (Edeme et al. 2020; Hernandez 
et al. 2020; Zhu et al. 2009). Furthermore, Fransen et al. (2019) highlight 
how transport disadvantage is often accompanied by unaffordable housing, 
causing long-term unemployment among people from disadvantaged regions 
to be more likely than among those from more prosperous regions. 

Empirical studies on socio-economic impacts in essence do not contra- 
dict in their research; however, they are limited in the sense that they often 
only explore one to three elements. Conversely, the studies that explore a 
variety of socio-economic impacts do so in the context of a specific type 
of infrastructure, such as transport, and thus ignore the potential effects of 
other types of infrastructure. In addition to these limitations, many studies 
do not differentiate the impacts on a regional scale, but instead provide an 
overarching understanding, often on a national or global scale. While such 
methods may work for certain countries where infrastructure investment and 
economic gains are more equally distributed, this is simply not possible for 
the UK with its dual economy, where regional inequalities already play a 
large role in infrastructure investment. Finally, the results from the various 
studies are produced using a wide range of methodologies, where different 
social impacts are analysed through differing means, and thus the conclusions 
drawn — though comprehensive — are not cohesive. 


e Government overly relies on cost (and to some extent time) to de- 
termine project success and frequently reduces contingency to ‘save’ 
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money — limiting room for manoeuvre if new information comes to 
light, or if economic conditions change. 

e Assurance (sometimes referred to as scrutiny) of projects often focuses on 
the process of completion against arbitrary budgets rather than the wider 
benefits that major projects can deliver in the long-term. 


The implementation of infrastructure investment is just as important as the 
investment itself. Governmental policies need to be effective in realising and 
distributing the benefits of infrastructure investment, or else risk adverse ef- 
fects, such as those of a study in Brazil that demonstrated infrastructure in- 
vestment having negative impacts on poverty alleviation due to inadequate 
and deficient public policy (Medeiros et al. 2020). Several studies report sim- 
ilar findings, where infrastructure-driven development in order to stimulate 
growth at a local level is stunted or not seen to its full potential due to public 
policies not directing or redirecting consistent resources efficiently (CECA 
2018; Edeme et al. 2020; Fransen et al. 2019; Hernandez et al. 2020; Torta- 
jada 2016). The UK in particular is facing several issues in successful policy 
implementation, due to lack of clarity and consideration for regional differ- 
ences, consequently receiving public backlash and overall public opposition 
to infrastructure projects (Coelho, Ratnoo and Dellepiane n.d.). 

In some cases, the adverse effects of infrastructure investment are inevita- 
ble, and it is up to governmental policies to efficiently mitigate these impacts. 
An example is soaring house prices creating a housing market bubble — a 
critical issue that many mega-cities such as London are facing today. Coelho, 
Ratnoo and Dellepiane (n.d.) demonstrate that, while several infrastructure 
investments have been made to disperse the population density, for example 
HS2 and Crossrail, these projects have once again received public opposi- 
tion due to the perceived policy bias that favours London and the Southeast 
region. Although there are numerous studies that investigate the impact of 
governmental policies on benefits realisation from infrastructure investment, 
many of them focus on developing countries that either do not have the foun- 
dation groundwork for infrastructure investment — making any large-scale 
project a heavy short-term financial burden — or they have a basis but lack 
development in current infrastructure which inhibits future projects. Finally, 
most of the empirical literature reviewed was published, or conducted their 
research prior to the impact of COVID-19, which has exacerbated inequali- 
ties between the North and South (Bambra et al. 2020). Thus, governmental 
policies now need to accommodate for the vulnerabilities that have been 
exposed due to decades of public spending cuts in the North, in addition to 
the already lacking infrastructure investments. 

The importance of delivering benefits from major project investments is 
perceived by the PACAC committee to be a “critical success factor”. Its re- 
cent report states that 


a project’s benefits are the very reason it is proposed and delivered, and 
the Committee does not believe that a project can be deemed successful 
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if it does not demonstrate realisation of its stated benefits. The Govern- 
ment has sought to justify spending millions of pounds on infrastructure 
during economically uncertain times by stating it will boost economic 
outcomes across the country. The Committee therefore expects the 
Government to be able to demonstrate growth because of this spend in 
future years. 


The question is, Who is tracking the benefits, and will there be a genuine ex 
ante evaluation of major investments in the decades to come? 


Case study 5: The Fujitsu Post Office Horizon 
Project 


In 1999, Fujitsu UK was awarded the contract to design, implement and 
operate a new information system infrastructure named ‘Horizon’ for 
Post Office Ltd — a retail post office company in the UK that provides 
a range of products including postal services and retail banking to the 
public through a network of branches. The company is wholly owned 
by ‘UK Government Investments’. ‘Horizon’ was a five-year project, 
completed in June 2001 at a total cost of c. £1bn. It was described at 
the time by Fujitsu UK, as “the largest non-military information tech- 
nology project of its kind in Europe”. The company’s own post-project 
marketing material lauded the “on time, to specification and within 
budget” achievements of the Horizon project. 

In subsequent years, sub-postmasters who used the Horizon system 
were blamed for unexplained accounting losses (known as balancing 
errors). Following investigations by Post Office Ltd, many were prose- 
cuted and subsequently tried and convicted of theft or false accounting 
offences in the criminal courts. In some cases, sub-postmasters were 
sentenced to prison, despite continued protestations of innocence. Dur- 
ing the period 2000-2015, over 900 sub-postmasters were prosecuted. 

It subsequently became known to Post Office Ltd that errors in 
the Horizon system may have been the cause of balancing errors, and 
that the subsequent prosecution of sub-postmasters on evidence gath- 
ered from the Horizon system may have been ‘unsound’. This led to a 
groundswell of public and political opinion that crystallised in a 2016 
group litigation in the High Court of Justice (Bates and others v Post 
Office Ltd). The judge, the Hon. Mr Justice Frazer, who had described 
the Horizon system as “extraordinarily complicated” found that the sys- 
tem contained “bugs, errors and defects” and that there was a “material 
risk” that balancing errors in the accounts were caused by the system. 
After the High Court ruling, the Criminal Cases Review Commission 
(CCRC) referred the cases of 42 former sub-postmasters to the Court 
of Appeal, which began taking evidence on 21 March 2021. The Post 


Estimation 121 


Office also settled a separate civil claim brought by over 550 claimants 
for £57.75 million, without admitting lability, in December 2019. 

The UK Government subsequently announced the scope of an In- 
dependent Review into the Post Office Horizon IT System and the 
criminal trials — following prime ministerial approval on the 26 Feb- 
ruary 2020. Darren Jones MP, the chair of the House of Commons 
(Business, Energy and Industrial Strategy Committee)’s inquiry into 
Horizon emphasised the importance of the inquiry in learning lessons 
from the project: 


It’s right that the Post Office has conceded, allowing the wrongful 
convictions of innocent sub-postmasters to be quashed. This is a 
landmark decision, but it is important the Independent Review has 
the powers it needs to play its part in getting to the bottom of this 
sorry story and help ensure it never happens again. This review is 
judge-led, which is welcome, but it needs to have the necessary 
powers to demand evidence and require witnesses to give evidence. 


It is quite possible that Horizon is responsible for one of the greatest 
miscarriages of justice in the history of the UK — only time will tell 
us this. The ongoing House of Commons (Business, Energy and In- 
dustrial Strategy) Select Committee enquiry into the Horizon project 
is stimulating a wider discourse on the lessons to be learned insofar as 
governance of IT projects are concerned. The reputational damage to 
a trusted ‘brand’ is also being assessed. The Post Office is perceived by 
many citizens in the UK as a ‘trusted’ organisation, but the events of 
the recent past are testing the public’s perceptions in a way that could 
never have been conceived before. 


5.6 Governments are experienced project promoters, 
but there is an absence of standardisation in the use 
estimating practices for major projects 


Estimating and forecasting are important underpinning principles in the 
practices used by government to make informed decisions on major projects 
and programmes. We have alluded to the impact of complexity in this chapter 
and elsewhere. The implications for estimating and forecasting in this con- 
text require an appreciation of the interrelationships between the technical 
and non-technical aspects — this is particularly important in the context of 
government, where departments must seek to co-operate in a holistic way. 
Sound estimating cannot be practised where disconnections between policy, 
project delivery and operational delivery plans occur. 

This is especially true when comparing major projects which exist in iso- 
lation from other projects and a consistent failure to learn lessons. The sector 
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has responded to the need to improve data capture analysis, retention and 
sharing; however, these efforts have not always been well-coordinated. There 
is no single standard for data collection, which makes records difficult to 
compare. This problem is compounded, to some extent, by a lack of trust 
and openness between project participants — prevalent among the causes of 
which are concerns about loss of competitive advantage, which often serves 
to hinder innovation and co-operation. 

There are few consistent approaches to data capture, risk or project man- 
agement in the work of government project delivery in the UK. Idiosyncratic 
methods adopted by contractors working on the same project — sometimes 
adopting different approaches — require that each major project is essentially 
being established from the ‘bottom up’ with an immediate need for data 
and systems integration from the outset. Several recently published standards 
have attempted to address this problem and their impact will become more 
apparent over time: 


e The ‘Functional Standard for Project Delivery’ was published by the 
government in 2019 to harmonise project delivery routines across 
departments. 

e The revised ‘Orange Book’ was published in 2020 providing guidance 
on risk and risk management in the context of the Green Book evalua- 
tion criteria for investments in projects and programmes. 

e ‘Cost estimating guidance’ was published by the Infrastructure and Pro- 
jects Authority in 2021 to promote ‘best practice cost estimating’ by out- 
lining the fundamentals that underpin a good cost estimate, the roles and 
responsibilities in producing, reviewing and owning the cost estimate to 
make informed decisions and the assurance process to ensure decision- 
makers use cost estimates in a responsible way. 

e ‘Guide for effective benefits management in major projects’ was pub- 
lished by the Infrastructure and Projects Authority in 2017 to provide 
structure and set expectations for major project teams when undertaking 
benefits management, and aligns with major project assurance processes 
and the ‘Green Book’ 5-case process. 


Whereas the UK Government has sought to improve standardisation in cost 
estimating practices to some extent, individual departments with significant 
project delivery responsibilities continue to maintain specific estimating 
expertise, the Cost Analysis and Assurance (CAAS) function of the Min- 
istry of Defence being one example. CAAS exists as the Mob’s ‘centre of 
excellence’ for pricing and costing support to some of the most complex 
defence engineering challenges, including the astute and successor submarine 
(‘Continuous At Sea Deterrence’) programmes and the Queen Elizabeth class 
aircraft carriers. Conversely, Highways England has instituted significant 
cost-estimation expertise to support the delivery of two strategic objectives — 
the Complex Infrastructure Programme (CIP) and the Road Improvement 
Strategy (RIS). 
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Case study 6: Highways England cost estimating for 
major projects 


Highways England Company Ltd is a wholly government-owned com- 
pany responsible for operating, maintaining and enhancing the motor- 
ways and major trunk roads across England. Its predecessor organization, 
the Highways Agency, was an executive agency of the Department for 
Transport before it was converted into a government-owned company 
on 1 April 2015. The Infrastructure Act 2015 requires Highways Eng- 
land to produce a long-term vision for what the strategic road network 
will require in terms of investment, and how money allocated to it by 
the Treasury will be invested in operations, maintenance, renewals and 
enhancements. This document is known as the Road Improvement 
Strategy (RIS). The current version, RIS2 (for the period 2020-2025) 
confirms a £27.4 billion budget and was produced following a staged 
process designed to ensure that investment decisions are based on ro- 
bust and rigorous analysis. The current RIS2 document identifies six 
strategic studies and a programme of refreshed route strategies cov- 
ering the entire network. The strategic studies focus on some of the 
greatest challenges confronting the road network, including routes into 
and around major urban conurbations in the North of England such as 
Manchester. In addition to the RIS, a separate portfolio of ‘Nationally 
Significant Infrastructure’ schemes are managed within the Complex 
Infrastructure Programme (CIP). 

The work of Highways England is regulated by the Office of Rail 
and Road (ORR). The origins of the ORR lie in the Office of Rail 
Regulation, created on 5 July 2004 by virtue of the Railways and 
Transport Safety Act 2003. It later became the Office of Rail and Road, 
responsible for overseeing Highways England under the Infrastructure 
Act 2015. As part of its regulatory duties, ORR commissioned an in- 
dependent review into Highways England cost estimating practices in 
2019, which included a focus on major projects. The findings from the 
report describe “consistent, templated cost estimation products with 
each prepared by its cost estimating team in collaboration with project 
teams for every major project and at every Project Control Framework 
(PCF) stage”, including: 


e Cost estimate summary sheet: containing a detailed breakdown 
of costs for development, land, preliminaries, construction, tax, 
utilities and supervision. 

e Range estimating template: to determine additional risks estimates 
and generate a three-point estimate profile for relevant items. 

e Estimate release form: essentially an assurance document to con- 
firm the procedural compliance and calculation accuracy. 
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The report also identifies three main estimating approaches, aligned 
with the relevant and sequential stages of the Project Control Frame- 
work, and described in the Highways England Cost Estimation Manual. 


1 First-principles, or ‘bottom-up’ estimating: Using detailed esti- 
mates for labour, plant and materials for each item of the works, 
typically based on a schedule of quantities and rates. 

2 Parametric estimating: Generally used to confirm ‘bottom-up’ es- 
timates, using the relationship between variables to calculate the 
cost or duration through statistical methods to determine cost esti- 
mating relationships. The independent report identifies a number 
of approaches by Highways England to reflect particular schemes, 
including ‘Smart Motorways’, bypass and widening schemes, and a 
preliminaries cost template. 

3 Analogous estimating: Used with historical data for similar pro- 
jects, and relevant where there is limited information. Analogous 
estimates are generally considered to be ‘top-down’ and therefore 
unlikely to be as accurate as other estimating techniques. 


The report concludes that there have been variances in costs across 
Highways England’s portfolio in recent years but does not attribute 
these to estimating inaccuracy. “Scope change’ and ‘external impacts’ 
are specifically identified as the primary cause of cost increases. 


5.7 Conclusion 


This chapter has explored the cost estimating paradox from the perspective 
of major project delivery in the UK through a series of interrelated problems 
illustrated by case studies drawn from the UK Government’s Major Projects 
Portfolio (GMPP). Initially, we explored the natural tendencies for humans 
to seek ‘certainty’ and the potential for important data to be disregarded in 
the decision-making process. Second, we problematised the need for, and the 
paucity of high-quality data at the front-end of a project by considering the 
tension that exists between the availability of information early in the project 
lifecycle and the necessity to quantify costs and benefits as part of the business 
case. Third, we considered the reasons as to why cost estimating is deter- 
ministically influenced, and the impact that this may have on the long-run 
perception of project success. Fourth, we posited that ‘the public gets what 
the public wants’ — a provocation that the public ‘at large’ tend to focus their 
attention on the ‘costs’ rather than ‘benefits’ of public spending — and thus the 
pressure to deliver ‘on time, on cost’ is, ostensibly, a pressure that is exerted 
chiefly by the public. This conundrum led us to seek to problematise the pro- 
ject performance versus project success tension from a wider socio-economic 
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perspective. Finally, we explored the absence of standardisation in the use of 
estimating practices for major projects and the implications that this may have 
for wider improvement to estimating practices across government projects 
and programmes. 


Note 


1 House of Commons Public Administration and Constitutional Affairs Commit- 
tee, Delivering the Government’s infrastructure commitments through major 
projects, Third Report of Session 2019-2021 Report, together with formal min- 
utes relating to the report, ordered by the House of Commons to be printed 23 
July 2020. 
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6 Incentives and politics 


The perverse incentives paradox: 
root cause of many other 
paradoxes; the case of the Dutch 
Betuweroute 


Bert van Wee 


6.1 Introduction 


The transport system in general, and, more specifically, transport infrastruc- 
ture projects as part of that system, has major impacts on society. On the 
positive side, it allows people to carry out activities in different places, and 
companies to transport goods. Without an adequate transport system, people 
could not easily reach jobs several (tens of) kilometres away from their place 
of residence; they could not easily reach hospitals, recreational facilities and 
many other destinations. In other words, the transport system provides ac- 
cess to destinations. That access contributes to the economy, and also to the 
well-being of people. Without an adequate transport system, goods transport 
costs (in terms of money, time and effort) would be way higher, negatively 
influencing the economy and strongly affecting the arrangement of produc- 
tion stages and trade patterns. 

On the downside, building infrastructure networks or projects, such as 
motorways or railways, costs a lot of money — in Western countries often in 
the order of magnitude of {10-30 million per kilometre — with added costs 
for maintenance and repairs. In addition, line infrastructures are barriers to 
animals and people, the use of infrastructures causing environmental im- 
pacts ranging from climate change due to greenhouse gas emissions to local 
noise and air pollution. It is also important to note that the pros and cons of 
transport infrastructure projects are not equally distributed across population 
groups and other actors. In general terms, the users of projects tend to benefit, 
while the non-users face disadvantages such as noise, exposure to pollution 
and barrier effects. Zooming out, the general taxpayer pays for the costs, 
whereas not all taxpayers benefit equally from transport infrastructure. Not 
only is the distinction between users and non-users relevant, but there is also 
a distinction between categories of users, for instance by income group or 
region. For example, high income segments of the population generally drive 
more by car, and fly more than low income segments. 

Consequently it is understandable that decisions on such projects often 
induce fierce debates, and it makes a lot of sense that many countries have a 
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ministry for transport, responsible not only for preparing decisions on invest- 
ments and maintenance of transport infrastructures but also for dealing with 
the downsides. 

On the face of it, one would expect decision-making processes in general, 
and on transport infrastructure in particular to follow a logical line of rea- 
soning, such as (1) assessing what the problem or challenge is, (2) listing can- 
didate solutions, (3) evaluating the pros and cons of candidate solutions and, 
finally, (4) making decisions. To support such a line of reasoning, several tools 
are available to assess the pros and cons of policy options (see Mouter 2020, 
for a recent handbook on the most common evaluation tools). Cost-Benefit 
Analysis (CBA) and Multi-Criteria Analysis (MCA) are the most commonly 
applied methods, other methods being Environmental Impact Assessment, 
Strategic Environmental Assessment and Cost-Effectiveness Analysis. How- 
ever, decision-making processes often do not follow the line of reasoning as 
presented above. Literature shows that the outcomes of CBAs on transport 
projects are poorly correlated with the final political decisions (Annema et al. 
2017; Eliasson et al. 2015; Odeck 2010). I do not want to argue that this 
definition is problematic per se. There could be good reasons for politicians 
to have preferences other than those implicitly suggested in a CBA. For ex- 
ample, a Green party may value environmental impacts more highly than 
assumed in a CBA. So the question of when a project is considered to be a 
success can be answered in multiple ways, depending on a person (politician 
or other) or organisation’s perspective and priorities. (See Chapter 2 of this 
book and the A303 example for a further discussion on what is ‘success’ in the 
case of transport infrastructure projects). As long as the problem of the chal- 
lenge is clear, the candidate solutions are listed clearly, and the pros and cons 
are clearly assessed, politicians can choose the solution they prefer, or decide 
to do nothing at all. This process allows for a mature and reasoned debate. 

On the other hand, a less explicit line of reasoning leaves room for per- 
verse incentives. Samset and Volden (2016) refer to the paradox of perverse 
incentives as follows: 


public investments with no financial obligations for the target group 
may cause perverse incentives and result in counterproductive projects”. 
They further explain: “The state often appears as a generous donor on 
behalf of taxpayers when financing projects that benefit specific groups 
or geographical regions. [...] When a project does not entail financial 
obligations for recipients, there is no incentive to opt for the most so- 
cially beneficial or cost-effective alternative. Different actors may have a 
vested interest in certain projects being chosen. [...] The term perverse 
incentives refers to the situation where one or more actors are motivated 
to make choices resulting in a project that is a complete failure seen in 
retrospect. 

(p. 308) 
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This chapter discusses the process leading to a positive decision to construct a 
controversial rail freight line connecting Rotterdam Harbour to the hinter- 
land, mainly the German Ruhrgebiet: the Betuweroute (also labelled as the 
Betuweline). 

It is not easy to be crystal clear in selecting the proponents of the Betuw- 
eroute, but key players are the Port of Rotterdam authorities, the (rail) goods 
transport division of the former Ministry of Transport, and interested groups 
of companies active in the transport sector. The Port of Rotterdam would 
undoubtedly benefit from the Betuweroute, because the connection between 
the port and the hinterland would improve, as a result of which the Port of 
Rotterdam would become more competitive relative to other harbours (Ant- 
werp, Le Havre and North German harbours). However, the Port of Rot- 
terdam would not have to pay for the line, so it is understandable that they 
wanted the Betuweroute to be built. For the same reason, it is understandable 
that interested groups of transport companies wanted the line: it would re- 
duce their generalised transport costs (if not, they would not use it), whereas 
they would not have to pay for the line. Note that the tariffs for using rail 
infrastructure do not cover all costs. EU regulations imply that the variable 
costs should be paid by the users, but not the fixed costs. The position of the 
goods transport division of the Ministry of Transport is a bit more difficult 
to understand. At the time, there were quite close relationships between that 
division and the goods transport sector. Of course the civil servants would 
not have to pay for the Betuweroute themselves. In the Netherlands, as in 
many other countries, there is competition not only between ministries in 
their claims for state money but also within ministries. My impression is that 
the goods transport division had an interest in attracting more money for 
goods transport infrastructure, in this case the Betuweroute, to increase the 
importance of that division. 

The methodology used for this chapter lies mainly in presenting a struc- 
tured impression based on long-term involvement of the author in debates 
related to the Betuweroute, research on its environmental impacts, and many 
contacts with politicians, interested groups, academics and others. Examples 
of this involvement include: a Dutch language article in a professional journal 
forecasting that the costs would be about four times higher than announced 
in the policy documents (Van Wee 1994); a study on the expected envi- 
ronmental impacts of the Betuweroute (Van Wee et al. 1994); supervising 
a PhD student studying cost overruns of large infrastructure projects in the 
Netherlands in general, and making detailed analyses of informal and formal 
decisions and related cost estimates of the Betuweroute (and also the High 
Speed Rail line from the Netherlands via Belgium to Paris) (Cantarelli et al. 
2010a); academic support of a committee advising on the role of parliament 
in decision-making on large transport infrastructure projects; and numerous 
contacts with the media. 

Section 6.2 describes the Betuweroute. The following section, 6.3, ex- 
plains what went ‘wrong’ in the process leading to a positive decision. This 
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section discusses the key issues in main lines. Section 6.4 elaborates on how 
this process can be explained, followed by Section 6.5, which zooms out, 
comparing the results with other projects and the literature. Section 6.6 links 
the perverse incentives paradox to other paradoxes presented by Samset and 
Volden (2016), arguing that the perverse incentives paradox helps to explain 
several other paradoxes. Some of the issues that went wrong, already intro- 
duced in Section 6.3, will be discussed in more detail, from the perspective 
of the additional paradoxes. Section 6.7 suggests what we can learn from the 
Betuweroute experience. Finally, Section 6.8 summarises the most important 
conclusions of this chapter. 


6.2 The case of the Betuweroute 


The Betuweroute is a 160 km long, dedicated rail freight line connecting 
Rotterdam Harbour to the hinterland, mainly the German Ruhrgebiet, and 
opened in 2007. The discussion on the potential construction of the rail 
freight line started in the mid-1980s, in a context in which the share of rail 
in goods transport in the Netherlands was quite small and declining. Be- 
tween 1970 and the mid-1990s, the share of rail freight had dropped from 
about 8% to below 5% of all goods transported in Dutch territory (KiM 
2016). Rail also played a small role in the transport of goods to and from 
Rotterdam Harbour. Germany had plans to invest billions of euros (Marken 
at that time) to improve the rail infrastructure connecting North German 
harbours to the hinterlands. By then, Rotterdam Harbour was the larg- 
est harbour worldwide (expressed in tons of goods transshipped). The fear 
of Rotterdam Harbour not being as competitive any more relative to the 
Northern German harbours, and maybe also to those of Northern France 
(Le Havre) and Belgium (Antwerp), probably played an important role in 
the debate. 

In addition, there were increasing concerns about the environment in gen- 
eral, both worldwide, as expressed by the Brundtland report, ‘Our common 
future’ (World Commission on Environment & Development 1987), as well 
as in the Netherlands. These concerns also applied to the environmental im- 
pacts of transport. Depending on the indicator used, the transport sector gen- 
erally had (and still has) a large share in the emissions of substances such as 
NO, and CO; and was a major source of noise nuisance. Per ton kilometre, 
road freight emissions were way higher than those of rail (Van Wee et al. 
1994), and a shift from road to rail was seen as an option to reduce the envi- 
ronmental pressure of freight transport. However, the rail transport system 
was generally considered to be not very mature, and on many tracks, rail 
freight was combined with passenger transport. This also applied to most of 
the rail to be transported from Rotterdam Harbour to the hinterland. The 
ambition of a strong increase in the volume of goods to be transported from 
Rotterdam Harbour to the hinterland raised concerns about the capacity and 
quality of the rail freight infrastructure. 
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Concerns about congestion also played a role. In the late 1980s and 1990s, 
congestion levels in the Netherlands were increasing (as in almost all West- 
ern and non-Western countries), and there were particular concerns about 
congestion levels on the motorways connecting Rotterdam Harbour to Ger- 
many. In the Dutch Second Transport Structure Plan (1990) specific cor- 
ridors considered to be vital for the Dutch economy were prioritised, the 
connection between Rotterdam Harbour and Germany being one such pri- 
oritised corridor. 

To summarise, for reasons of the economy (including congestion) and the 
environment, the quality of rail freight infrastructure connecting Rotter- 
dam Harbour and Germany was considered to be problematic. The debate 
started with a solution, the Betuweline, without making clear as to what it 
was a solution for, other than in quite general terms, as explained above. The 
first argument used by proponents was an economic argument: the Dutch 
economy would benefit from building the Betuweroute. When the CPB, 
the Netherlands Bureau for Economic Policy Analysis, wrote a report that 
the Betuweroute would probably not be economically viable (CPB 1993), the 
argument that the environment would benefit became more important in 
the debates. However, a study on the environmental impacts of the Betuw- 
eroute (Van Wee et al. 1994) revealed that the impact of building the Betu- 
weroute on Dutch freight emissions would be very low (1%—2.5%) or could 
even be absent if there was no high level connection to the German rail 
freight system (the decision of Germany to build their part of the line was 
not yet made). Emission reductions could only be realised when additional 
rail freight was mainly the result of a shift from road to rail. But in the 
economic underpinnings of the Betuweroute, proponents argued that the 
rail line would attract additional freight via the Netherlands, at the cost of 
Belgium and German harbours. It was argued that of the additional goods 
transported via the Netherlands (due to the Betuweroute), 50% would not 
be transported via the Netherlands (but via neighbouring countries), 40% 
would be transported via barge, and only 10% would be transported via road 
(Van Wee et al. 1994). This explains why the report on the environmental 
impacts of the Betuweroute argued that, in this case, the environmental ben- 
efits would be negligible: the substitution from road to rail would be very 
limited. So, either the Betuweroute was to some extent beneficial for the 
economy (mainly because more goods would be transported via the Neth- 
erlands), but then there were no environmental benefits, or vice versa. In 
addition, if there were environmental benefits, the cost-effectiveness of the 
Betuweroute would be extremely low (Van Wee 1994). Later it was argued 
that the Betuweroute would reduce congestion on the competing motorway 
(from Rotterdam to Germany), but traffic experts expected this reduction 
to be marginal. Note that the demarcation for all calculations was the Neth- 
erlands, both in the case of the economy and the environment. It could be 
that increased emissions on Dutch territory, because additional freight would 
be attracted, would lead to a reduction in emissions on German or Belgian 
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territory. Of course, comparable distribution effects apply to economic im- 
pacts, so the demarcation to focus on Dutch territory for both the environ- 
mental and economic effects was at least consistent. 


6.3 What went ‘wrong’? 


From the perspective of ‘rational decision-making’, several things went 
‘wrong’, leading to a positive decision to build the Betuweroute. As argued 
above, I assume ‘rational decision-making’ to at least include the following 
steps (see also Chapter 3): 


e a clear description of the problem or challenges 

e a selection of candidate options 

e an ex ante evaluation of the pros and cons of candidate options 

e a decision including the rationale relative to the three previous points. 


I realise there are many other, often more complicated, process models, but 
these four steps are, in my opinion, the minimum requirements. 

Below I briefly list several violations of these four steps, returning to sev- 
eral of the points made in more depth in Section 6.6, where I compare the 
process leading to a positive decision to build the Betuweroute to several of 
the other paradoxes presented in Samset and Volden (2016). 


1 There was no clear debate on the problems/challenges/aims. 

In informal debates, proponents argued that the capacity of the hin- 
terland rail connection was simply too low to accommodate the fore- 
casted demand, so more capacity was needed. The debate started with 
a solution, the Betuweroute, without a clear analysis of the challenges, 
problems or aims. As explained above, there were some notions with 
respect to the economy, the environment and congestion, but these were 
formulated in quite general terms. Moreover, the topics were considered 
specifically as far as they were relevant to the transport of goods from 
Rotterdam Harbour, but not in general terms. For example, there were 
no debates on the general performance of the Dutch economy, and its 
strengths and weaknesses, leading to the conclusion that lack of capac- 
ity in the rail infrastructure for the transport of goods from Rotterdam 
Harbour to the hinterland would play a significant role in the economic 
performance of the Netherlands, and that improving the rail freight sys- 
tem was to be prioritised over any other ‘solutions’. 

Obviously key actors wanted the Betuweroute for whatever reason(s), 
and were not interested in the rationale of the aims, problems or chal- 
lenges underpinning the project. From their perspective, this makes 
sense: if one wants something, a debate that could challenge that desire 
is not attractive because it could lead to other solutions, or even to not 
taking any action at all, because no solution could be a ‘good’ solution for 
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the problem or challenge. For a wider discussion on the front-end stage 
of projects: see Chapters 3 and 4 of this book. 
2 No alternative solutions were considered. 

Also as explained above, no alternative solutions were considered, such 
as inland shipping or road related solutions. This despite several claims in 
the debates that there was much surplus capacity in the barge system, and 
high quality waterway connections of Rotterdam Harbour to the hin- 
terland. The Netherlands is located on the sea side of several large rivers, 
providing very good access by barge to many destinations in Northwest- 
ern Europe, especially in the Netherlands and Germany. Also the road 
network, particularly the motorway network, is very dense in the Neth- 
erlands, so several alternatives to rail transport already existed, and new 
infrastructure or the reduction of bottlenecks in existing infrastructure 
would increase access via roads or waterways considerably because of the 
availability of already existing high level networks. 

In addition, for the dedicated rail freight line, only one alternative was 
proposed: the line as it was built. There was only one additional option: 
a split of the line on Dutch territory, including a rail freight line to the 
north of Germany. However, that option was considered to be not real- 
istic, and was soon abandoned. 

Again, as for topic 1, this made sense from the perspective of those 
interested in the Betuweroute, because a debate on alternative solutions 
to the aims, problems or challenges could lead to alternatives such as 
measures in the area of inland shipping, which were considered to be not 
attractive by the proponents of the Betuweroute. 

3 The cost estimates were unrealistically and deliberately low, and the de- 
mand forecasts were flawed. 

In the Second Transport Structure Plan (STSP), published in 1990, the 
Betuweroute was announced at a cost of £1.1 billion (2.3 billion guilders 
at the time, the conversion rate being 2.2 guilders per euro). For a 160 
km rail line, with many barriers such as a channel, in a densely populated 
country, this was a ridiculously low estimate, not only from hindsight 
but back in 1990. I made a back-of-the-envelope calculation based on the 
costs of a few other large transport infrastructure projects, and estimated 
the costs to be about £4.5 billion (10 billion guilders) (later published 
in Van Wee 1994). In 1990/1991, I needed a more adequate estimate 
for the development of long-term scenarios for Dutch society, published 
in 1992 by the CPB. On ‘phoning a contact for the Betuweroute at the 
Ministry of Transport, asking for a realistic cost estimate, I was shocked 
to receive the reaction: the line could definitely be built for £1.1 billion, 
and it was absurd that I had some doubts. My objection that the estimated 
costs per kilometre would be way lower than the costs of other recently 
built transport infrastructure projects was waived away: the ministry was 
sure that the line could be built for £1.1 billion. I then realised that 
something was wrong, and this was politically very sensitive. I checked 
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my calculations, asking a few other researchers for feedback, and they 
confirmed my estimates. I published my concerns in 1994 in an article 
in a professional journal (Van Wee 1994). By then I was working for a 
governmental institute closely related to policy making, the National 
Institute of Public Health and the Environment (Dutch abbreviation: 
RIVM), which later became the Netherlands Environmental Assessment 
Agency (Dutch abbreviation: PBL). All the research we did was as input 
for policy making, but was not allowed to draw conclusions on what 
policy makers should or should not do. Because my role was that of an 
independent researcher, not an activist trying to influence decisions, and 
not that of a researcher working for a university (and thus at a greater 
distance from policy making), I took no further action. 

Later an official of the ministry who was heavily involved in policy 
making for the Betuweroute told me she did not sleep at all for two nights 
after I published the article, because she was afraid politicians would read 
it, and the positive decision to build would be endangered (which did not 
happen). The final costs of the Betuweroute were £4.7 billion (Cantarelli 
et al. 2010a). It seems that my estimate had been very accurate, but note 
that due to inflation between 1994 and 2007, £4.7 billion at 2007 prices 
represents a lower value than at 1994 prices. 

Also the demand forecasts were not at all realistic (Meijdam 1993), but 
more wishful thinking. In the community of transport economists, the 
demand forecasts made on behalf of the ministry were considered to be 
ridiculously high, while those in the scenario without the Betuweroute 
were considered to be ridiculously low. Consequently the impact of the 
Betuweroute on rail freight demand, and the likely economic benefits of 
the line were heavily overestimated. 

Several political parties supported the line at a very early stage, without 
any information on the problems/challenges, alternatives and effects. 

Before the formal decision to build, several political parties commit- 

ted to the Betuweroute, without having a clear picture of the problems/ 
challenges and alternatives, or their pros and cons. In the Dutch context 
(and in many other democracies), it is difficult to change a political po- 
sition after early commitment: politicians who do this get the reputation 
of being unreliable. This early commitment led to the installation of a 
committee to advise on the role of parliament with respect to large infra- 
structure projects (TCI, Temporary Committee Infrastructure projects) — 
see below. One of its recommendations was to not support or reject pro- 
jects or proposals before key process steps (as presented above) have been 
made. 
Two metres of reports were published preceding the decision-making, 
but many of those were manipulated or biased. The reports also made 
different assumptions with respect to the macro-economic scenarios 
when estimating the impacts of the line. No clear report summarising 
the main pros and cons was published. 
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The previous topic does not suggest there was no research on the im- 
pacts of the Betuweroute. The formal decision to build up all the reports 
together resulted in a pile 2 metres high. One of the problems was that 
this was way too much for politicians to read/check, even though the 
reports were published over several years. Another problem was that the 
assumptions behind the estimates were different, inconsistency between 
the claimed economic benefits (as a result of attracting more goods trans- 
port via the Netherlands) and the claimed environmental benefits (due 
to a shift from road to rail) being an example. A third problem was that 
several studies were not conducted according to academic standards, and 
were far from neutral. Several reports were written with the a priori idea 
that these should support a positive decision to build the Betuweroute. 
Academics criticised some of those reports heavily, but their criticisms 
generally did not reach politicians. A final problem was that there was 
no document giving a systematic and clear overview of all the relevant 
pros and cons of the Betuweroute, let alone possible alternatives. Several 
former members of parliament told me later that they missed a clear and 
neutral document of only a limited number of pages that summarised the 
rationale for the Betuweroute as well as providing an overview of the 
pros and cons. 

6 There was no systematic evaluation of all the pros and cons (CBA or 
MCA). 

Consequently any systematic comparison of the pros and cons of build- 
ing the Betuweroute, such as a cost-benefit analysis or multi-criteria 
analysis, was lacking. The reports written only provided pieces of the 
jigsaw puzzle, explaining the effects, but the pieces did not fit, nor did 
they give the complete picture. An ex ante evaluation of the pros and 
cons of the alternatives (in this case, at least, to build or not to build the 
Betuweroute, preferably also of alternatives) would lend discipline to the 
debate. First of all it would lead to the development or application of fu- 
ture scenarios that are internally consistent. In case of the Betuweroute, 
as explained above, the assumptions behind the economic argumentation 
were inconsistent with the implicit assumptions underpinning the en- 
vironmental argumentation. Second, at least for CBAs, there is quite a 
mature debate on how CBAs should be carried out, and to some extent 
this also applies to the state of knowledge in the 1990s. It should hence 
have been easier to trace manipulation. Third, such an ex ante evaluation 
would probably have received a lot of attention from the media, inter- 
ested groups (for and against the Betuweline) and academics, and thus it 
would have fuelled the debates. 

7 There was no idea about opportunity costs. 

In most countries, the budgets of ministries are relatively stable. This 
implies that the yearly budgets for new transport infrastructure projects 
do not vary much between years. This means that projects that are fi- 
nanced come at the cost of other projects that could have been built. In 
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economic terms, building a project implies opportunity costs: an oppor- 
tunity to build one or more other projects is missed, and so are the related 
benefits of those projects. It was not clear at the costs of which projects 
building the Betuweroute did come. It might have been at the cost of 
other projects that were never realised, or were postponed. 

This problem was not limited to the Betuweroute only, because in the 
years when it was constructed, other large-scale transport infrastructure 
projects were built, such as the high speed rail line connecting Amster- 
dam and Rotterdam to Antwerp, Brussels and Paris, and extensions of 
the port of Rotterdam. Those actors with an interest in smaller trans- 
port infrastructure projects often complained that these large projects 
took too large a share in the overall budget of the Ministry of Trans- 
port for transport infrastructure projects. It is important to realise that 
in the Netherlands, local municipalities and provinces have limited self- 
generated financial resources. Consequently they rely on financial re- 
sources provided by the Ministry of Transport (at national level). 

8 To increase support, many additional measures to reduce local environ- 
mental impacts were taken. 

The original plans for the Betuweroute included hardly any measures 
to reduce the local impact, such as noise walls or barriers. I cannot prove 
it, but I always had the impression that this was done deliberately to 
achieve as low as possible cost estimates. But that was not at all realistic. 
For example, in 1978 a law had already been implemented to reduce ex- 
posure to high noise levels of road and rail projects. The early designs of 
the Betuweroute did not fulfil the related legal requirements for noise. 

Local resistance against the Betuweroute was fierce: many local mu- 
nicipalities protested because they did not benefit from the line, but did 
face negative local impacts such as noise, barrier effects and visual im- 
pacts. To reduce local resistance, many local measures were added to the 
original plan. These measures were partly needed for legal reasons (noise) 
but more measures were added. No analysis of the additional costs and 
benefits of those added measures was made. 


6.4 How can this be explained? 


An important question is: how can what went ‘wrong’ be explained? Samset 
and Volden explain this, emphasising the differences between aligning the 
recipients’ objectives and national objectives, and information asymmetry. 
Both explanations definitely apply to the Betuweroute. As explained above, 
the proponents had specific objectives based on self-interest, rather than na- 
tional objectives. They also had way better insights than parliament on the 
pros and cons, based on an overload of reports, that were largely not neutral. 

But there is more to add to the explanation of what went wrong. The area of 
political economics is very useful in helping to understand what went wrong. 
I depart from the political economics framework for the implementation (or 
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Figure 6.1 A political economy model of transport innovations. 


not) of candidate innovations, as proposed by Feitelson and Salomon (2004) 
and visualised in Figure 6.1. 

The framework is very useful to understand the success or failure of com- 
plex candidate innovations that need the support of both public and private 
actors, such as large-scale transport infrastructure projects. In most countries 
such projects are publicly financed. Even in countries like France, where 
parts of the transport infrastructure are privately financed (toll roads), support 
of the public sector is needed. 

I use this framework to help understand what went wrong in the main. 
Figure 6.1 shows that candidate innovations need to be technically and polit- 
ically feasible. The Betuweroute is a conventional rail line, so technical feasi- 
bility was not at all a point of discussion. Figure 6.1, in addition, shows that 
for political feasibility, social feasibility is also important. Although there was 
substantial resistance from local municipalities and other interest groups along 
the proposed route, overall, the social support was quite strong. Awareness 
of the increasing congestion on motorways and the environmental pressure 
of road transport fuelled this support, and was dominant in the sanctioned 
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discourse (see Figure 6.1). The perception was that a dedicated rail line from 
Rotterdam Harbour to the German Ruhrgebiet would be an effective solu- 
tion to these problems. In addition, several actors, especially organisations 
such as Rotterdam Harbour, and transport interest groups, supported the rail 
line. A factor less visible in the framework is that in the preceding decades, 
rail investments had focussed on passenger transport, not freight transport. 
This, combined with the low share of rail in freight transport on Dutch ter- 
ritory resulted in a positive attitude of many actors towards investing in a rail 
freight line, that is, the Betuweroute. 

Looking to what went wrong, I will next discuss some possible explanations. 


1 There was no clear debate on the problems/challenges/aims. 
2 No alternative solutions were considered. 

Those actors with an interest in the rail line, such as Rotterdam Har- 
bour, and highly placed people working for the former Ministry of 
Transport on rail freight had no interest in first discussing the precise 
problems and alternative solutions. They wanted the Betuweroute, and 
were able to put it on the political agenda at an early stage of the discus- 
sions. The decision-making procedures (see Figure 6.1) did not ask for a 
clear assessment of the problems/challenges, nor for alternative solutions. 
In other words, the perverse incentives paradox very well describes what 
went wrong in these respects. 

3 The cost estimates were unrealistically and deliberately low. The demand 
forecasts were flawed. 

Comparing the final costs with those on the decision to build, cost 
overrun appeared very limited, but as explained above, comparing the 
final costs with the original cost estimates published in 1990, the cost 
overrun was very large (Cantarelli et al. 2010a). This is not uncommon: 
the vast majority of large transport infrastructure projects worldwide face 
cost overruns (Flyvbjerg et al. 2014), the strategic behaviour of those 
having an interest in a positive decision to build being a likely explana- 
tory factor (Cantarelli et al. 2010b). 

The problem with strategic behaviour and cost overruns is that those 
who benefit from a positive decision to build are not those who need 
to pay. It is the tax payer who pays, or — if one assumes the total public 
budget for transport infrastructure to be constant — those who would 
have received the benefits from projects that could not be built. Because 
it is seldom made explicit which those projects are, those who lose out are 
set to remain anonymous. 

Comparable problems exist with respect to demand forecasts. 

Figure 6.1 explains that perceived problems and the distribution of 
cost and benefits influence social feasibility. The congestion and envi- 
ronmental problems were widely shared, but due to the notion that the 
costs would be quite low, the distribution of costs and benefits hardly 
played any role in the early stages of the debate. 
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Several political parties supported the line at the very early stages, with- 
out any information on the problems/challenges, alternatives and effects. 
In line with the previous point, several political parties committed at 
an early stage, probably because they thought the Betuweroute would 
be an interesting option to reduce congestion and the environmental 
problems of road freight, and would consequently be socially feasible. 
Probably they did not worry too much about the distribution of costs 
and benefits because of the low cost estimate overall. Next they thought 
it would be a nice strategic decision to improve the competitiveness of 
Rotterdam Harbour, and this position was probably fuelled by input 
from industry (companies in the area of goods transport and Rotterdam 
Harbour) who suggested there would be a problem with the competi- 
tiveness of Rotterdam Harbour if the Betuweroute were not to be built 
(‘perception of problems’, in Figure 6.1). They were probably influenced 
by actors having an interest in the early commitment of politicians. 
Two metres of reports were published preceding the decision-making, 
but many of those were manipulated or were at least biased. Moreover, 
the reports made different assumptions with respect to the impacts of the 
line. 
There was no systematic evaluation of all the pros and cons (CBA or 
MCA). 

At the time, there was no tradition of systematic evaluations on the 
pros and cons of candidate large transport infrastructure projects. Hence 
it was relatively easy to order studies and reports that considered only a 
part of the questions to be answered. It was also relatively easy to select 
non-neutral research institutes which wrote reports that the ministry 
wanted to be written. The early commitment of several political par- 
ties (see above) resulted in support for the positive attitude of successive 
Ministers of Transport, and because of that early positive commitment, 
the manipulated reports did not receive much attention from politicians. 
The decision-making procedures (see Figure 6.1) did not prescribe how 
research needed to be done, or how assessments of the pros and cons 
needed to be made. 

There was no idea about opportunity costs. 

By then (and still) the concept of opportunity costs hardly received 
any attention in debates on large infrastructure projects — debates were 
mainly on projects in isolation — in terms of the decision-making pro- 
cedures. The Betuweroute debate was no exception. Discussions on op- 
portunity costs would have extended the debate relative to considering 
alternatives to the Betuweroute, and the main actors did not have an 
interest in such wider talk. The decision-making procedures (see Figure 
6.1) did not ask for opportunity costs to be made explicit. 

To increase support, many additional measures to reduce local environ- 
mental impacts were taken. 
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Because of the early commitment of several political parties and the pos- 
itive attitude of successive Ministers of Transport, opposition needed to 
be avoided as much as possible, or, formulated in terms of Figure 6.1, 
social feasibility was ‘bought’. Therefore, if local municipalities protested 
against the Betuweline because of local impacts, the reaction was often to 
reduce such impacts by adding hardware to the rail line. 

To summarise, those with an interest in a positive decision to build the 
Betuweroute did not have to pay its costs, and had no interest in a more 
mature procedure as suggested above (the problems/challenges, options, 
evaluation). In addition to this discussion based on the framework of 
Feitelson and Salomon, public choice theory (Buchanan 1986) is helpful 
in understanding the positive decision to build the Betuweroute. Public 
choice theory explains political behaviour from an economic perspec- 
tive, assuming the self-interested behaviour of people. Politicians want 
to be (re)elected. They receive more attention by playing a role in clearly 
visible large-scale projects than, for example, in reducing the mainte- 
nance problems of infrastructure, or from supporting smaller, less visible 
projects, even if these have better benefit to cost ratios. Flyvbjerg et al. 
(2014) argued that politicians have more to gain from a positive decision 
to build large-scale infrastructure projects, than to lose from the cost 
overruns of such projects. 

Specifically for the perverse incentives paradox introduced above, 
explanations are first that the Port of Rotterdam supported the project 
but did not have to pay, so it was easy for them to ask for the line. Sec- 
ond, the ministry probably had as a perspective: the budget for transport 
infrastructure is constant. Which project to choose? Because passenger 
transport then dominated expenditure, those who were involved with 
goods transport also wanted some money to be spent on goods transport 
infrastructure. 


6.5 The broader context: other projects and literature 


This section elaborates on the question as to what extent the results can be 
transferred to other contexts. The overall picture is that the Betuweroute is 
in some respects a ‘worst in class’ example, but it is not unique. 

As explained above, the Betuweroute faced a strong cost overrun if we 
compare the final costs with those in the Second Transport Structure Plan. 
Cost overruns are by no means an exception to the rule, the strategic be- 
haviour of those with an interest in a positive decision being one frequently 
suggested explanation (Cantarelli et al. 2010b; Flyvbjerg et al. 2014); (see also 
Chapter 5 of this book). Perverse incentives apply in the sense that those re- 
sponsible for unrealistically low cost estimates do not pay the bill, literally and 
figuratively. Note that cost overruns also have other explanations (Cantarelli 
et al. 2010b), optimism bias being an important one. Optimism bias implies 
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that those estimating costs did not intend to underestimate them, but were 
too optimistic. If optimism applies, one could argue that there is still a prob- 
lem with ‘wrong incentives’ because there is a lack of incentive to do ‘better’. 
Those estimating costs could be provided with guidance on how to estimate 
costs via institutional rules. A specific method to arrive at cost estimates is 
Reference Class Forecasting (Flyvbjerg and COWI 2004). This method im- 
plies that cost estimates of comparable projects realised in the past are used to 
estimate the costs of new projects. In addition, it is an option to allocate some 
of the risks of ‘wrong’ cost estimates to private parties. 

Next, overly optimistic estimations on the use of transport infrastructure 
projects are also not an exception: many ex ante evaluations overestimate 
demand (Van Wee 2007). Institutional rules could provide incentives to do 
‘better’. 

Looking at other projects, the most comparable project was the already 
introduced high speed rail line from the Netherlands to Belgium and Paris, 
completed in 2009. Comparable problems existed with respect to cost over- 
runs, demand shortfall and decision-making. In addition, after the opening 
of the line, there were problems later on with the poor quality of the trains, 
and currently (2021) the line has only limited use, way less than forecasted / 
assumed in the decision-making processes. 

Another example is a not (yet) built fast rail line (conventional rail, HSL 
or Maglev) connecting the North of the Netherlands to the Randstad area 
(Amsterdam and Schiphol Airport). The North of the country is doing eco- 
nomically less well, at least in terms of unemployment rates, average incomes 
and diversity of the economy. About two decades ago, discussions started on 
building a fast rail line to reduce these problems. At an early stage, some kind 
of commitment was given to the Northern provinces to build the line. Two 
CBAs were carried out, both showing that the costs would be much higher 
than the benefits. Related travel demand modelling studies showed that 
trains would have increasingly lower occupancy rates on the more northerly 
sections of the line. Economic impacts would be limited, and the increased 
number of jobs in the North would not come at the cost of the prosperous 
West, but at the cost of the Eastern part of the country, also a region that is 
not doing well economically. Partly based on the CBA results, a negative 
decision was made, and the North received other forms of financial com- 
pensation. Nevertheless the debate was re-opened, and a recent study was 
carried out, a CBA being part of that study. Again the conclusion was that 
costs would be way higher than benefits (about 2.5 times higher, at least). So, 
again, this is an example of a discussion in which only one type of solution 
was proposed — a fast rail line connecting the North of the Netherlands to 
the Randstad area. In this case, multiple alternatives to that rail line were 
considered, such as fast conventional rail versus high speed rail. Again, the 
proponents of the line did not have to pay for it, and the perverse incentives 
paradox describes what happened. 

Zooming out to the much wider category of Dutch transport infrastruc- 
ture projects, the way these are financed is of interest. As already explained 


Incentives and politics 145 


above, local and provincial authorities have relatively low ‘own income’ to 
spend on transport infrastructure projects compared to other countries. They 
largely rely on money from the national Government. This has important ad- 
vantages, because the national Government evaluates candidate policies from 
a national perspective, not only a local or regional perspective. The risk of 
departing from the local perspective is that a project could make sense from 
a local perspective, but not from a national perspective, because the gains for 
a local municipality come at the cost of losses to other municipalities. For 
example, improving the canal connecting Amsterdam harbour to the North 
Sea could be beneficial from the Amsterdam perspective, but not from the 
national perspective, because Amsterdam and Rotterdam Harbour compete. 
A disadvantage of the reliance on national funding is that local governments 
could ask for projects they would not build, and would not have to pay for 
fully, even if they had the money. After all, they receive most — if not all — of 
the benefits, but pay only a small part of the costs. On the other hand the 
national perspective is an example of a ‘good incentive’, at least from the per- 
spective of the Dutch tax payer. 


6.6 Relationships with other paradoxes 


As explained above, this paper departs from ‘the perverse incentives’ para- 
dox. The paper of Samset and Volden (2016, p. 299) presents ten paradoxes. 
The ‘perverse incentives paradox’ (Paradox 9) influences several of the other 
paradoxes, at least Paradoxes 1, 2, 3, 4, 6, 7 and 10. In this section, I will ar- 
gue how. The definitions of all paradoxes below are literally taken from the 
original paper of Samset and Volden (2016). 


1 The success paradox: success is measured in terms of tactical performance 
rather than strategic performance. 
Samset and Volden make clear that success is a difficult term, and can 
be defined and operationalised in many different ways. They further 
explain: 


...1s necessary to distinguish between the projects’ tactical and stra- 
tegic performance. Success in tactical terms typically means meeting 
short-term performance targets, such as producing agreed outputs 
within budget and on time. These are essentially project manage- 
ment issues. Strategic performance, however, includes the broader 
and longer-term considerations of whether the project would have a 
sustainable impact and remain relevant and effective in its operational 
phase, throughout its lifespan. This is essentially a question of get- 
ting the business case right, or, in short, of choosing the most viable 
project concept. [...] Strategic performance ... includes the broader 
and longer-term considerations of whether the project would have 
a sustainable impact and remain relevant and effective in its opera- 
tional phase, throughout its lifespan. [...] Strategic performance is 
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a question of how the project performs after the outputs have been 
delivered. 
(p. 300) 


Distinguishing tactical versus strategic performance makes sense, and the 
paradox suggests the focus is often on tactical, not strategic performance. 
In case of the Betuweroute first of all, to the best of my knowledge, it 
was not made explicit what ‘success’ would mean, not even in tactical 
terms. There were notions of a shortage of capacity on the existing rail 
tracks for goods to be transported from Rotterdam Harbour to Germany. 
In addition there were abstract notions that the Betuweroute would be 
a ‘strategic decision’, and good for the Dutch economy, but it was less 
clearly explained what strategic performance was to be obtained. This can 
be understood from the perspective of the paradox of perverse incentives: 
if strategic performance was specified, if would be relatively easy to show 
that it was doubtful that the Betuweroute would be a success. If success 
was defined in quantitative economic terms, the report of CPB (1993) re- 
ferred to above would undermine the claim that the Betuweroute would 
be a success. 

As Samset and Volden make clear, sustainability should be a compo- 
nent of strategic performance. In the case of the Betuweroute, environ- 
mental impact was the most likely sustainability dimension, in addition 
to economic performance. Indeed, as explained in Section 6.3, environ- 
mental impacts were mentioned in the Betuweroute debates. However, 
it is striking to see the line of reasoning in the arguments. In 1994, I was 
asked by the Dutch Ministry of Housing, Spatial Planning and the En- 
vironment (a ministry that does not exist anymore in its current form) 
to write a report, together with others, on the average freight emission 
factors for rail, road and barge. My contact at the ministry told me that 
the results would be used to quantitatively underpin the plea for building 
the Betuweroute, from an environmental perspective. Seemingly there 
was an interest in measuring strategic performance. But the problem was, 
as explained above, that only providing emission factors would be mis- 
leading, because the Betuweroute was, for economic reasons, argued to 
attract more freight to be transported via the Netherlands and to com- 
pete with barge transport. So suggesting that the Betuweroute would be 
good for the environment because emissions per ton kilometre of rail 
transport were lower than those of road transport, without including 
the impact of the economic scenarios (transport volumes by mode with 
and without the Betuweroute) on emissions would be incomplete, or 
even misleading. Therefore, we, as researchers, decided to multiply the 
emission factors by transport volumes for the scenario with and without 
the Betuweroute, showing that the impact of the Betuweroute on freight 
emissions would be very low. This was despite the request, and even or- 
der, to leave out these calculations. The report hardly gained attention in 
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the debate, but later on several politicians told me they felt manipulated 
because as far as they could remember, they had never explicitly received 
information other than the emission factors. Moreover, as one politician 
told me, maybe our report had been provided on a Friday afternoon, to- 
gether with many other documents, to be read over the weekend, when 
family members expected him to spend time with them, so he could have 
easily overlooked the report. 

As a side note: the calculations we made by multiplying average emis- 
sion factors for road, rail and barge and transport volumes was a very 
rough method. As Van Wee et al. (2005) argue, average emission factors 
do not necessarily apply to specific cases, for several reasons. For exam- 
ple, the characteristics of goods vary by mode, so for the transport of a 
given type of goods, other factors could apply. In addition, detour factors 
differ between modes because of different networks, and long distance 
road transport has lower emission factors than average road transport. 

All in all, the perverse incentives paradox explains the occurrence of 

the success paradox in the case of the Betuweroute: those with an interest 
in a positive decision to build the Betuweroute had no interest in discuss- 
ing the strategic performance of the rail line. 
The paradox of the significance of front-end management: less resources 
are used up front to identify the best conceptual solution (project gov- 
ernance), than to improve tactical performance during implementation 
(project management). 

Samset and Volden put this paradox in the context of uncertainty and 
information to reduce uncertainty, and argue, 


It is widely believed that uncertainty is highest at the initial stage, 
when the project concept is conceived, and that it tends to reduce 
rapidly as information accumulates over time. [...] Major issues such 
as agreeing on the most effective solution to a problem and the choice 
of concept need to be dealt with as early as possible — later on is too 
late. [...] Where projects fail strategically, it is likely that the problem 
can be traced back to decisions in the earliest phases, when the initial 
idea was conceived and developed. What happens during the front- 
end phase is therefore essential for a project's success. [...] In most 
cases the key issue at the earliest stage is to shed sufficient light on 
the underlying problem that provides the justification for the project, 
and the needs that the project is meant to satisfy. [...] It is a paradox 
therefore that most of a project's planning resources may be spent on 
detailed planning and engineering, while too little is usually spent 
on getting the idea right from the start where the potential to reduce 
uncertainty by means of adding information is the largest. The para- 
dox is that most resources are used to reduce uncertainty during the 
implementation phase, where the potential is much less. 

(p. 301) 
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In case of the Betuweroute, a lot of documents had already been pro- 
duced years before the decision to build, but as explained above, there 
was no information on what the problems or challenges were, and which 
alternative solutions would be available. The information aimed to un- 
derpin the need for the Betuweroute and to increase the likelihood of a 
positive decision to build. So again those having an interest in a positive 
decision to build, including the Ministry of Transport and Public Works, 
did not have an interest in a strategic discussion to identify the prob- 
lems, challenges and alternative solutions. In other words, the perverse 
incentives paradox helps to explain what happened in the case of the 
Betuweroute with respect to the paradox of the significance of front-end 
management. 

3 The paradox of early information overflow: decisions are based on masses 
of detailed information up front rather than carefully selected facts and 
judgemental information relevant to highlight the essential issues. 

Samset and Volden write about this paradox, 


the front end phase is when fundamental choices are made, uncer- 
tainty is at its highest freedom to choose is at its optimum, and avail- 
able information is most restricted. Adding information, therefore, 
makes sense — but only to a certain degree. The crucial issue is not 
the volume but what type of information is needed. |...] In the initial 
phase of a project the priority is to establish an overall perspective, 
and to analyse the problem in its context, considering the needs and 
priorities of stakeholders, users and affected parties, in order to come 
up with a sensible strategy. Opportunities and risks should be consid- 
ered. Experience suggests that creativity, imagination and intuition 
can be more valuable at this stage than large amounts of data. 


They also emphasise 


the need to invest in relevant information at the earliest stage of a 
project, while at the same time limiting the search to what is useful 
for decision-making at this stage. A targeted search for information 
regarding the main uncertainties likely to affect the project is more 
cost-effective than an unguided search, since it makes it possible 
to increase the share of relevant information and reduce the total 
amount. 


(p. 302) 


This paradox definitely also applies to the Betuweroute. As explained 
above, 2 metres of reports were produced, but not the type of reports 
needed in line with the argumentation of Samset and Volden. The re- 
ports aimed to convince decision-makers and others of the need for a 
positive decision to build the Betuweroute. Again the perverse incentives 


Incentives and politics 149 


paradox helps to explain what happened in case of the Betuweroute with 

respect to the paradox of early information overflow. 

The paradox of the opportunity space: the choice of conceptual solution is 

made without systematically scrutinising the opportunity space up front. 
Samset and Volden write about this paradox, 


Every project is initiated to solve some problem or meet some needs. And 
every project faces a choice of concept in terms of how to solve this prob- 
lem. Consequently, a key task in the early phase of a project is to identify 
possible ways to solve the problem it has been mandated to solve (setting 
up the opportunity space), furthermore to evaluate alternative concepts 
(limiting the opportunity space), and decide on the one best suited. There 
is much evidence to suggest that this is not always how things are done. 
(...). A (...) case study of 23 major public investment projects (Whist and 
Christensen, 2011) went deeply into how the analytical and political pro- 
cesses interacted during the front-end phase, in order to understand how 
this affected the outcome of the projects. It was found that the majority 
of projects started out with a predetermined solution. 


(pp. 302-303) 


The Betuweroute perfectly matches the pattern found by Whist and Chris- 
tensen (2011): as explained above, the project started with a predetermined 
solution. No alternatives were considered. A difference between the Betu- 
weroute and the first part of the description of Samset and Volden is that 
there was not a clear description of the problem. Moreover, the need was 
formulated in a narrow-minded way: there was a need for a better rail con- 
nection between Rotterdam Harbour and the hinterland. But the need 
behind that better connection was not made explicit. Again the perverse 
incentive paradox helps to explain what happened: if those with an interest 
only want a predetermined solution, then it is better to avoid any discussion 
on the problems it should solve or the needs it should satisfy. 
The cost estimation paradox: the focus is on the final cost estimate (the 
budget), while early cost estimates are overlooked. 

Samset and Volden write, referring to a report written by Welde et al. 
(2014), 

The report discusses possible reasons for the substantial underes- 
timation in early phases. An often used distinction is made between 
political, technical, and cognitive reasons (e.g. Flyvbjerg, 2005). It 
may be very difficult to prove that the cause is political, but in sev- 
eral of the projects there were clear indications that the first estimate 
was deliberately low in order to increase the chance of the project 
idea being considered. This corresponds well with other studies that 
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have attempted to prove that costs are underestimated deliberately to 
make the projects appear more attractive. (...) Hence, it is clearly a 
paradox that so little attention is devoted to the initial estimate. 

(p. 306) 


As explained above, the early costs estimates of the Betuweroute were ri- 
diculously low, and discussing the estimates was a very sensitive matter. 
I will now elaborate a bit more on the dynamics of the cost estimates. 
Cantarelli et al. (2010a) introduce the concept of lock-in to explain what 
happened. Following Woerdman (2004), they explain (p. 793), “Lock-in 
is created when suboptimal policies are used as a consequence of path de- 
pendency, even though a better alternative is present”. They explain how 
low cost estimates might lead to a situation of lock-in, and how politi- 
cians and other actors might decide to support a project long before the 
formal decision to build. They propose a theoretical framework to help 
understand which factors (‘input variables’) lead to lock-in (Figure 6.2). 
It is beyond the aim of this chapter to fully discuss the framework, but it 
shows that lock-in occurs due a complex interplay of factors leading to esca- 
lating commitment, the need for justification, inflexibility and the closure 
of alternatives. The framework helps in understanding the importance of 
the cost estimation paradox: cost estimates that are too low might lead to 
early commitment, long before the formal decision to build. Actors with an 
interest in a specific project might benefit from the communication of low 
cost estimates at an early stage, in the hope of early commitment. 

Cantarelli et al. show the timeline of cost estimates in the years preced- 
ing the formal decision to build (Figure 6.2). 


Input variables Indicator Outcome 
Investments D  Sunkcosts > 
(ime,costs) fe 
i 
1 
1 
1 
Social pressures > H 
1 
Need for 1 > 
Face-saving > Justification H 
l Lock-in 
Agreements A 1 (decision-making 
and project level) 
Solution starting point y > Escalating --- > 
commitment > 
Strategic behaviour A 
Political vulnerability 
Inflexibility, 
Path dependency > Closure > 
alternatives 


Figure 6.2 Theoretical framework for lock-in. Solid lines represent the influence of 
conscious lock-in, dotted lines the influence of unconscious lock-in. 
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as a starting point Aaroeine is of A 
(secalating conaiment) Naw (escalating commitment) ý Decision to 
S track law build 
Mty] rail [ l 
Years: 1990 1991 1992 1993 1994 1995 1996 1997 
Costs *: 1,134 2,335 2,832 3,239 3,744 4,034 4,138 4,243 


* Official reported budget including inflation (in billion euro) 


Figure 6.3 Timeline for the Betuweroute project (decision-making level). 


Figure 6.3 shows that the initial cost estimate was as low as £1.1 billion, 
whereas the cost estimate at the decision to build was £4.1 billion. It is 
difficult to say at what point of time politicians and other actors had the 
idea that there was no longer any way back, for reasons of early commit- 
ment. The research team (Cantarelli and colleagues) tried to empirically 
assess these actor specific points in time by contacting the politicians in- 
volved in the decision-making process. The responses they received were 
so low that they had to abandon the study. 

Again the pattern is in line with the paradoxes discussed above: the 
paradox of perverse incentives explains that it made sense for proponents 
of the Betuweline to come with excessively low cost estimates. Next a 
process of lock-in occurred, as a result of which there was no way back, 
long before the decision to build. The theoretical framework presented 
by Cantarelli et al. (2010a) might be helpful to understand the process 
of lock in. 

Not only is the discussion on lock-in and early commitment relevant 
to help understand the importance of the perverse incentives paradox 
and the impact it has on the cost estimation paradox, but it also makes 
clear that the literature on cost overruns comparing final costs with 
costs at the decision to build shows only part of the real problem. Be- 
cause early cost estimates tend to be lower than those at the decision to 
build, and because of early commitment (as a component of lock-in), 
it is very possible that cost overruns based on a comparison of final 
costs and costs at the ‘point of no return’ are way larger than those that 
follow from comparing the final costs with those communicated at the 
decision to build. 

The paradox of disregarded analyses of costs and benefits: detailed esti- 
mation of cost and benefits is commonly done up front, but disregarded 
by decision-makers, who tend to emphasise other aspects. 

Samset and Volden write about this paradox, 


A substantial amount of resources is devoted in major investment 
projects to establish a decision basis. Detailed Cost-Benefit Analyses 
are often performed, and complex models are developed to simulate 
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traffic volumes and other inputs to these analyses. However, there 
are indications that decision-makers have little confidence in Cost 
Benefit Analysis in Norway. [...] The paradox in this case is that so 
much effort is devoted to the calculation of a net present value that 
decision-makers may not find useful or credible. 

(pp. 306-307) 


As explained above, no CBA preceded the decision-making process in 
the case of the Betuweroute. The perverse incentives paradox probably 
explains why: a CBA would show that the costs would be way higher 
than the benefits. So the proponents had no interest in a comprehen- 
sive, clear and neutral CBA. It is important to realise that at the time of 
the Betuweroute debate, it was not common to carry out a CBA. But, 
as will be explained in the next section, the discomfort that many had 
with respect to the whole Betuweroute procedure and debate led to the 
obligation to conduct a CBA for large transport infrastructure projects 
in the Netherlands. 

On a side note: about a decade or so ago, I was contacted by an or- 
ganisation that wanted to organise a conference on the positive decision 
to build the Betuweroute. This was after the opening of the line in 
2007. They suggested the core should be a debate between a proponent 
of the Betuweroute and someone who was ‘against’ building it. They 
contacted me to fulfil the role of the latter. I first explained that I was 
not ‘against’ building the Betuweroute, but supported the idea that par- 
liament should be well, clearly, impartially and timely informed. This 
did not apply to the Betuweroute. Second, I explained that it was very 
unlikely that the benefits could ever be high enough to compensate the 
high costs, and that we did know this at an early stage. So, I was willing 
to make these points, and debate with a proponent. I even made a ‘quick 
and dirty’ CBA underpinning my second point. The organisation was 
pleased with my willingness to contribute. Several weeks later they 
contacted me again: the conference had been cancelled because none 
of the proponents was willing to debate with me on this topic at a con- 
ference. This gave me the impression that they had already known that 
the costs would be larger than the benefits, and that parliament had not 
been informed adequately. 

7 The paradox of ‘predict and provide’: the tendency is to choose a ‘predict- 
and-provide’ strategy rather than to explore alternative solutions. 

Samset and Volden write about this paradox: 


When confronted with capacity problems, the planners, who are 
often engineers, almost always recommend increased capacity based 
on estimates of future demand. However, unsurprisingly, there is 
often excess demand for public services and infrastructure offered 
free-of-charge to citizens. The need should not be defined narrowly 
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as a need to increase capacity but rather as a need to solve the conges- 
tion problem. The latter allows for a variety of measures, including 
demand regulation, congestion pricing, and legal and informative 
measures, most of which are far cheaper than a construction project 
to expand capacity. [...] The paradox in this case occurs when needs 
and benefits assessments in public infrastructure projects are decou- 
pled from overriding political priorities and goals, possibly because 
such overriding societal goals are conflicting and multidimensional. 


(p. 308) 


This definitely applied to the Betuweroute debate. A manipulated fore- 
cast was made showing a very strong increase in rail freight demand, 
and that there was definitely not enough capacity on the existing rail 
infrastructure. Proponents hence argued that the line needed to be built. 
The variety of other measures as addressed by Samset and Volden was 
ignored. The perverse incentives paradox clearly explains that this makes 
sense from the perspective of the proponents of the line: if you want a 
line, you do not want a debate on alternative solutions for rail freight 
transport capacity; neither do you want a debate on the overriding politi- 
cal priorities and goals, in this case relating to the economy, environmen- 
tal impacts of freight transport, or congestion on the road connection 
from Rotterdam Harbour to the hinterland. 


6.7 What did we learn? 


Because of the positive decision to build, combined with a lack of under- 
standing of the rationale and no clear and neutral assessment of the pros and 
cons of the line, many people (politicians, interested groups, citizens, etc.) felt 
manipulated, although on the positive side, the process also contributed to 
important improvements in decision-making procedures. 

First of all, together with the process leading to the construction of the 
High Speed Rail line from the Netherlands via Belgium to Paris, which faced 
comparable criticism, it led to the development of a manual to ex ante eval- 
uate large transport infrastructure projects, the core being the obligation to 
carry out CBAs for such projects according to the manual (Eygenraam et al. 
2000) published by the Netherlands Bureau for Economic Policy Analysis 
(CPB). The report was the result of many debates and contributions from 
leading researchers and institutes. The manual not only prescribed a CBA 
for large national transport infrastructure projects; it also explained how to 
carry out CBAs. In addition, after three years of applying the manual, a thor- 
ough evaluation of experiences was made, leading to several additions to the 
manual. Up to 2017, over 100 CBAs had been carried out using the manual 
(Annema et al. 2017). 
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Second, in 2013 a more generic manual was written, to be applied to all 
kinds of projects, and not only transport projects (Romijn & Renes 2013), 
published by CPB and PBL (Netherlands Environmental Assessment Agency). 
Again, a CBA is the default, and instructions were given on how to ex ante 
evaluate projects. 

Third, largely because of the experiences with the role of parliament in the 
decision-making process of the Betuweroute and the High Speed Rail line 
from the Netherlands via Belgium to Paris, as explained above, a commit- 
tee (temporary committee infrastructure projects — Dutch abbreviation TCI) 
was founded to study and advise on the role of parliament in the decision- 
making processes of large infrastructure operations. The cost overruns of 
both projects played an especially important role in the establishment of the 
committee. The committee report (Tweede Kamer 2004) advised, among 
other things, on research support for parliament, removing the ban on con- 
tacts between members of parliament and civil servants, and making sure that 
members of parliament are not overloaded with less relevant information, 
while important information should be sent to parliament. 


6.8 Conclusions 


This chapter departs from what Samset and Volden (2016, p. 308) refer to as 
‘the paradox of perverse incentives’: 


public investments with no financial obligations for the target group may 
cause perverse incentives and result in counterproductive projects. [...] 
The term perverse incentives refers to the situation where one or more 
actors are motivated to make choices resulting in a project that is a com- 
plete failure seen in retrospect. 


The chapter has explained that the positive decision to build the Dutch Betu- 
weroute, connecting Rotterdam Harbour to the German Ruhrgebiet can be 
understood by this paradox. Although a cost-benefit analysis was not carried 
out, it is extremely likely that the benefits will never be high enough to 
compensate for the construction costs of almost £5 billion (even ignoring 
other costs, such as maintenance costs). No private party had an interest in 
participating in the building of the rail line. Yet a positive decision to build 
was made. Those actors supporting the construction of the line did not have 
to pay for it — the Dutch tax payer had to pay. However, the social benefits in 
terms of the environment and congestion reduction are very small. As econ- 
omists have argued: the main beneficiaries are German companies who have 
an additional transport option to receive and ship goods way below the real 
transport costs. What is more, these German companies did not pay anything 
for the construction of the line, as they do not pay taxes in the Netherlands. 

The chapter also makes clear that many things went wrong, such as having 
no clear explanation of the problems the line was supposed to solve, or the 
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related challenges. No alternatives were considered, and the cost estimates 
were seriously flawed. 

I argue that the perverse incentives paradox helps to explain several of the 
other paradoxes discussed in the paper of Samset and Volden: those having an 
interest in a positive decision to build the Betuweroute did not have to pay 
for it, and this resulted in several perverse incentives that are further detailed 
and discussed by other paradoxes. 

Although the Betuweroute is a “worst in class’ example, it is no exception. 
Cost overruns and demand shortfalls are a worldwide problem. The problem 
that those who benefit from a positive decision to build do not pay for the 
costs, providing a wrong incentive, is also common. 

On the positive side, the Netherlands learned a lot from the Betuweroute 
(and the HSL South). Since 2000, it has been obligatory to assess candidate 
national large transport infrastructure projects via a social cost-benefit anal- 
ysis, making use of a dedicated manual. Based on the advice of a special 
temporary committee report published in 2004, the role of parliament in the 
decision-making process was strengthened. Later (2013) a manual for the ex 
ante assessment of all kinds of projects was introduced. Several experts think 
that the decision not to build a fast rail line connecting the north of the Neth- 
erlands and the west was largely due to the CBA showing that the costs would 
be way higher than the benefits. Koopmans (2010) estimated the CBA related 
costs of CPB, and assumed that a negative decision to build candidate projects 
evaluated according to the CBA manual for which costs exceeded benefits 
could be attributed to 10% of the CPB and CBAs. The welfare losses (calcu- 
lated from the difference between estimated costs and benefits) were about 
£20 billion. Hence, the positive value of the CPB and CBAs is estimated to 
be £2 billion (10% of £20 billion), implying a benefit-cost ratio of the related 
work of 17. In other words, the obligation to carry out a CBA according to 
the CBA manual was a decision which significantly increased welfare in the 
Netherlands. The negative experiences with respect to the Betuweroute, the 
lessons learned and the actions taken made positive contributions to Dutch 
society. 

A final remark on the paradox, as formulated by Samset and Volden (2016, 
p. 308), in particular the sentence, “The term perverse incentives refers to the 
situation where one or more actors are motivated to make choices resulting in 
a project that is a complete failure seen in retrospect’: I argue that this applies 
to the Betuweroute, but not only in retrospect; long before the formal deci- 
sion to build, several researchers had already warned that a positive decision 
would result in welfare losses. 
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7 Closing the loop 


Ex ante and ex post evaluation in 
order to learn from mistakes and 
successes 


Knut Samset and Gro Holst Volden 


Paradoxical dilemmas such as those discussed in this book could have been 
avoided in many cases, if planners and managers were better at learning from 
experience. There are valuable lessons to be learnt from success as well as 
failure — in many cases, lessons that are quite obvious and relevant. How- 
ever, ex post evaluation for designated learning purposes is not commonly 
performed. The tendency is not to look in the mirror to learn from similar 
undertakings, but only straight ahead on how to plan the next big project. 
This predicament was discussed in our initial paper on the ten paradoxes — in 
fact, we considered presenting it as Paradox 11. 


7.1 On evaluation in general 


Generally speaking, the purpose of evaluation is threefold, that is, it con- 
tributes to efficient control, management and learning. In some cases the 
focus is on all aspects simultaneously; in other cases it is restricted to one, for 
instance the control aspect, in which case the exercise comes close to what is 
normally termed an audit. The control aspect is the most restricted, focusing 
on expenditure in relation to budget, progress in relation to plans, outputs in 
relation to standards, etc. The management aspect is broader, looking at per- 
formance, organisational issues, processes, etc. The learning aspect of eval- 
uation is even broader and requires a more open-ended mandate in order to 
focus on and get a deeper understanding of causes and effects, achievements 
seen in relation to experience with similar projects, etc. It is often necessary 
to study groups of several projects simultaneously in order to draw lessons for 
the future (Samset 2003). 

An investment case, process or project is typically divided into three dis- 
tinct phases. At the beginning, the idea and decision phase lasts until the final 
decision to implement is made. The implementation phase follows, until the 
project’s outputs are realised. Finally, there is an operational phase, in which 
the benefits of the project are realised or revenue comes in. This process is 
illustrated in Figure 7.1. 

Experience indicates that today, most evaluation activities occur during 
the implementation phase or just after its conclusion, with options designated 
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Figure 7.1 Evaluation of an undertaking at different points in time. 


interim evaluation and final evaluation, respectively. This is puzzling, be- 
cause the implementation phase is the period in which the project is least 
likely to benefit from an evaluation (Samset 2003). An interim evaluation 
can help to avoid or correct mistakes during the implementation phase, that 
is, it provides management information. A final evaluation assesses the re- 
sults at the conclusion of the implementation phase, that is, it provides control 
information. 

It is a paradox that systematic ex ante and ex post evaluations are rarely 
used. Ex ante evaluation provides strategic information about the main choices 
at an early stage, when the possibility to influence the course of an under- 
taking is greatest. It aims to find the best approach or conceptual solution of 
possible alternatives, and to clarify the major questions that will determine 
the terms of planning, which are essential. Ex post evaluation undertaken 
well into the operational phase will provide learning information to improve 
design and decisions for similar projects in the future, which is also valuable 
(Andersen et al. 2008). 

Ex ante evaluation consists of a broad initial assessment aimed to identify 
which alternative will yield the greatest benefit from an intended investment. 
More commonly, considerable resources are used on detailed planning of one 
specific solution, whereas alternatives are not considered, or are inadequately 
assessed early on. Consequently, there is no adequate basis for concluding that 
the preferred alternative is the best choice. 

The benefits to be accrued from comprehensive appraisals upfront were 
demonstrated by a World Bank evaluation of more than 1,000 investment 
cases. It concluded that as many as 80% of the cases that conducted a thor- 
ough feasibility study and secured the ‘Quality at Entry’ were successful, 
whereas only 35% of those with poor preparation upfront were successful 
(World Bank 1997). 
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The benefit of ex ante evaluation is principally related to whether one can 
identify the best solution to the problem at hand and avoid expensive and 
ineffective solutions. This will be based on estimates of the project’s effects. 
Such estimates are useful for management decisions during implementation, 
and provide benchmark information for interim and ex post evaluations. Ex 
ante evaluation may also be useful for studying different scenarios and the 
effects of changes in certain parameters during implementation. Systematic 
sensitivity analysis is all too little used, even in major, extensive projects 
(Jovanovic 1999). 

The purpose of ex post evaluation is to draw lessons of experience from 
relevant cases. The motivation is principally that it may contribute to double- 
loop learning. Consequently, to evaluate a single project is seldom sufficient; 
it is necessary to evaluate several similar projects. However, this is not com- 
monly done; therefore, the use of evaluation for learning purposes is not 
encouraging (Schindler & Eppler 2003). 

Businesses, particularly industries, are better than the government at both 
ex ante and ex post evaluation, with their ex ante market forecasts and ex 
post user surveys. However, these are relatively limited assessments in which 
return on investment (ROI) is the paramount objective, and market demand 
and user satisfaction are key evaluation criteria. In such cases, a narrow eco- 
nomic incentive clearly motivates the studies. 

Each investment case or project represents only one of several possible con- 
cepts that may be realised. In advance, other concepts may have been assessed 
but rejected in favour of the one preferred. After the project is completed, it 
is evaluated in relation to planned and projected goals and effects, and to the 
ex ante situation — but rarely in relation to the counterfactual, that is, what 
the situation would have been had the project not been realised (Harberger 
1997). The reason for this is first and foremost that people’s preference is 
commonly to invest in something new. However, experience shows that, 
in retrospect, the zero option might have been the most beneficial choice. 
Therefore, an ex ante evaluation should also review the zero option because 
it affords a basis of comparison for assessing future benefit. 

Incentives in public investment projects are not as clear. The goal is often 
more compound and complex. Attention is focused on effects for users and 
society, and on benefit-cost efficiency, but individuals are not made account- 
able to the same extent in relation to the achievement of objectives, as is the 
case in business. Consequently, there is no clear tradition for such evaluations. 
One notable exception is the Norwegian Ministry of Finance, which intro- 
duced a scheme in 2000 for ex ante evaluation of the country’s largest public 
investment projects, the so-called quality assurance (QA) scheme (Volden and 
Samset 2017). Twenty years later, more than 300 projects have been exposed 
to this QA scheme. In the coming years, several ex post evaluations will be 
made of these projects, under the auspices of the Concept Research Pro- 
gramme at the Norwegian University of Science and Technology (NTNU). 
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The study presented in this chapter is based around the first 29 such evalua- 
tions undertaken to date (2021). 


7.2 Evaluation based on insufficient information 


In an ex ante evaluation, much must be based on assumptions, because fewer 
facts are available. It is more bounded by history, facts and interpretations, 
leading to a selection of decision premises influenced by organisational struc- 
tures and actors’ roles, as noted by Simon (1957). Lack of information leads 
to greater reliance on experience, on opinion, or, at worst, on guesswork. 
This is a disadvantage but not a hindrance. A combination of facts and well- 
founded assumptions are the best you can provide in the early phase. How- 
ever, an ex ante evaluation can contribute systematic generation and analysis 
of such information. The assumption is that this process will provide better 
results than no systematic analysis, although the information base is weak. 
There is a large amount of literature in this area, for example, Goodwin and 
Wright (1991), Bazerman (1994) and Williams et al. (2009). 

As already mentioned, ex ante evaluation occurs when principal decisions 
are made and the possibility of making changes is greatest — but also when 
uncertainty is highest and the information basis is most limited. What mat- 
ters then is which type of information is needed. Since the main focus is on 
the problem to be solved and corresponding needs to be met, there is less need 
for detailed scrutiny of the alternative solutions to the problem. 

This illustrates a dilemma, because most projects start out with only one 
specific conceptual solution to a problem. Much of the information generated 
is associated with that particular solution. Indeed, in many cases, the amount 
of specific, detailed information contributes to restricting the original choice 
of concept to the extent that it will eventually be the realised option. This is 
referred to as Paradox 3 in the ‘Paradoxes’ paper. In too few cases are other 
possible concepts identified and analysed sufficiently. 

Concerning the information needs in ex ante evaluation, the amount of 
information generated often outstrips the needs — one asks for more, although 
the needed information is already there (Feldman & March 1981). Experience 
suggests that in the earliest parts of the process, our concern should be to un- 
derstand the problem in its context, together with the needs and interests of 
affected parties, in order to design a sound solution in strategic terms and to 
assess possibilities and risks. Therefore, at the outset, the need is first and most 
importantly to establish perspective. Intuition, creativity and imagination 
can, then, be more valuable than exact, quantitative information. One could, 
in many cases, turn the problem around and argue that rather than being a 
hindrance, lack of detailed information early on can actually be a benefit, 
providing focus and flexibility to the analysis. 

Moreover, the validity of information may be a problem. It is obvious, 
as experience confirms, that the more precise the information is, the more 
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rapidly it becomes obsolete. It is tempting to speak of information half-life. 
For example, in a rapidly expanding market, the value of information on 
demand as a basis for planning may depreciate within a few days. At the same 
time, the validity of qualitative measures is often more durable than precise 
quantitative information. Consider, for example, the basic perceptions of the 
needs of user groups. We may know little about the urgency of the need, 
but can be certain that it will persist for a long time. Therefore, it may be 
included in initial assessments. 

Omitting details and less relevant information helps to avoid analysis paral- 
ysis. This is yet another argument for not swamping the initial process with 
detailed quantitative information. The need for precise and detailed informa- 
tion increases with the advance of the process. Later, such information will 
be more readily available (Williams et al. 2009). 

Commonplace opinion holds that the quality of a decision base is crucial 
for decisions. However, opinion varies on the meaning of ‘the quality of a de- 
cision base’. Experience suggests that decision-makers are often less affected 
by decision bases than one would think is desirable. Studies have shown that 
even when good decision bases are available, decisions are more influenced 
by the decision-maker’s intuition and personal or political preferences than 
by facts and analyses (Feldman & March 1981; Henden 2004; Mintzberg 
2000). The practical implications of this are not necessarily disastrous. Several 
studies have shown that in many situations, intuition is preferable to rational 
analysis, particularly when the decision situation is complex, as is often the 
case in large investment projects. The assumption, then, is that intuition is 
based on experience and training. If the decision-maker has the necessary rel- 
evant professional experience in the sectors involved, intuition can help make 
sound decisions more rapidly. However, we cannot expect effective intuition 
without thorough knowledge of the theme at hand. In turn, that character- 
istic involves many years of experience, combined with workable analyses. 

Early on, it is essential to establish the best possible understanding of reality 
as a basis for identifying a suitable strategy. Subsequent critical steps include 
identifying the overall framework conditions that should guide subsequent 
decisions on the choice of concept, and then the necessary framework condi- 
tions that should guide the planning and shaping of the project. The frame- 
work and conditions form the specific solution to the problem in hand — in 
other words, what the project is to deliver. In such a stepwise process, it is 
advantageous to choose an approach with corresponding increases in degree 
of detail and the level of precision of information. The challenge is to acquire 
the essentials and limit the magnitude of what is communicated. This im- 
proves communication and increases the likelihood of the evaluation results 
being used. The Pareto Principle, also known as the 80-20 rule, may be used 
to illustrate information needs. The notion is that in a cause-effect relation- 
ship, a few vital causes lead to the greater part of the consequences. To clarify 
these causes is the central challenge. 
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An evaluation involves an assessment employing specific evaluation criteria 
and measures against standards or expected values, such as knowledge and 
proficiency relative to learning goals or effects relative to strategic plans. A 
key question in an evaluation of a public investment project is whether the 
project was successful. Evaluation is thus a matter of determining the degree 
of success. 

Samset (2003) argues that in order to be truly successful, public projects 
must not only perform well operationally but also tactically and strategically. 
However, whereas operational project success is highlighted by practitioners 
as well as academics, tactical and strategic success is often ignored, possibly 
because it challenges the way analysts think and has political aspects to it 
(Samset & Christensen 2017). 

All projects are explicitly or implicitly based on an assumed set of causal 
relationships between inputs, project activities, outputs, outcome, and ulti- 
mately, broad societal outcome, as illustrated in Figure 7.2. Several authors 
argue the merits of using this so-called logic model (McLaughlin & Jordan 
1999; Samset 2003), also referred to as the programme theory (Chen 1990; 
Rogers et al. 2000; Weiss 1997) as representation of the project to help visual- 
ise important aspects, and especially when preparing for an evaluation. The 
logic model helps to clarify for all stakeholders: the definition of the problem, 
the overarching goals, and the relationship between project success on an 
operational level (output), tactical level (outcome) and strategic level (achieve- 
ment of the societal objective). 

There are many ways to evaluate, depending on the type of project, pur- 
pose, tradition, etc. A much-used evaluation model is based on five evalu- 
ation criteria that together express the degree of success on all three levels. 
This standardised set of five evaluation criteria is used by the UN and other 
institutions and development aid organisations, and has been endorsed by the 
OECD-DAC (OECD 1991, 2002). 

Evaluation according to this model highlights (1) the need for the project 
(relevance), (2) whether the uses of resources and time are reasonable (effi- 
ciency), (3) whether expectations are fulfilled (effectiveness), (4) what other 
positive or negative effects may occur as a result of the project (impact) and 
(5) whether the positive effects persist after its conclusion (sustainability). 

As noted by Picciotto (2013), development projects are not so different from 
projects in developed countries. The five criteria reflect hard-won lessons of 
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Figure 7.2 The project’s logic model. 
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Figure 7.3 Evaluation is part of an information process based on overriding questions 
and, through analyses and decision-making, results in actions at the detail 
level. 


experience and have, by and large, replaced prior approaches that focused 
only on inputs and outputs. They can be used equally at project, programme 
and policy level, and are aligned with the results-oriented stance favoured by 
most countries. Other sectors have introduced variants of the criteria (see, for 
example, the European Commission (2013) concerning socio-economic de- 
velopment in Europe; ALNAP (2006) concerning humanitarian projects; and 
the European Commission (2015) concerning regulations). The framework 
has hardly changed over time and thus shows an impressive stability. 

Evaluation typically involves a process in which overriding evaluation cri- 
teria are disaggregated into more detailed evaluation questions relevant to the 
conditions to be evaluated. Then, information is acquired and aggregated to 
support conclusions relative to the overriding evaluation criteria. 

The OECD-DAC model is an example of a goal-oriented evaluation, 
which is one of many possible approaches. The model is principally intended 
for application in the ex post situation, but could conceivably be even more 
beneficial in the front-end phase. The question is to what extent sufficient 
information exists at an early stage. 


e Regarding the issue of efficiency, doubtless the costs of a project and the 
nature of its delivery are reasonably well understood at an early stage. 
However, there may be doubt as to whether the cost estimates are re- 
alistic and the conditions of implementation will allow that outputs are 
produced as anticipated. Consequently, gauging efficiency is hardly 
worthwhile in the earliest phase, while the complications facing planners 
and decision-makers in estimating realistic costs clearly indicate that the 
basis for evaluating efficiency is usually poor. 

e The same could be true of effectiveness. Undoubtedly, the first-order ef- 
fects sought are usually clearly known, but realistic forecasting can be 
notoriously flawed. 
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e Early estimates of impact are even more difficult. Undoubtedly, experi- 
mental knowledge may be acquired by studying similar projects, but we 
face conditions that are difficult to forecast and arguably require imagi- 
nation and guesswork beyond our capabilities. 

e However, the situation for relevance differs. Common sense and user sur- 
veys, as well as knowledge of markets, laws and regulations permit us to 
form an early, accurate picture of whether an initiative is relevant. That 
we may be notoriously poor at this sort of early evaluation is not due to 
it being impossible, but rather to it not being done to a sufficient extent. 

e Initially, forecasting future sustainability is also difficult. However, the 
question is closely related to whether an initiative is relevant. Moreover, 
we can usually realistically analyse cashflow early on. 


The conclusion, then, is that with reasonable effort, we can obtain a good 
picture at an early stage of whether a proposed project is relevant and sustain- 
able, whereas the other three criteria must be assessed at a later stage in the 
planning process. The good news is that relevance and sustainability are pre- 
cisely the two aspects that are most crucial to whether a project will succeed 
strategically. Consequently, such a delimitation of a very first ex ante evalu- 
ation will, therefore, be the minimalist answer to what might be an appro- 
priate approach, not least because the benefit relative to resource allocation 
will be quite high. Results from ex post evaluations carried out according to 
the same criteria will thus come into use at different stages in the planning 
of new projects. 


7.4 Ex post evaluation of large public projects. The case 
of Norway 


In Norway, as mentioned above, ex ante evaluation is compulsory in a scheme 
requiring external quality assurance of the decision basis for projects with an 
estimated investment cost exceeding 100 million euros. The scheme includes: 
(1) appraisal and external quality assurance of the choice of conceptual solu- 
tion before the Cabinet’s decision on whether to start a pre-project, and (2) 
subsequent quality assurance of the project management basis and cost esti- 
mate before the project is submitted to Parliament for approval and funding. 
Quality assurance is performed by external experts who are pre-qualified by 
the Ministry of Finance (Volden & Samset 2017). 

As of today, (2021) about 300 projects have been subject to this scheme, of 
which some 120 have so far been completed. There is strong evidence that 
early evaluation and quality assurance has improved the Norwegian gov- 
ernment’s basis for decisions regarding major public investments (Kvalheim 
et al. 2015) and that most of the projects keep within their budgets (Welde 
2017). Nevertheless, projects also need to be evaluated ex post, to verify how 
they perform strategically. To that effect, we applied a broad, goal-oriented 
evaluation framework based on the one presented above on 29 Norwegian 
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Figure 7.4 The Norwegian evaluation format. 


projects that had been quality-assured in their front-end phase. The presenta- 
tion below is an update of the findings in Volden (2018a) based on the first 
20 of these projects. 


7.5 A format of six evaluation criteria 


The chosen evaluation format is depicted in Figure 7.4, and is based on the 
five OECD-DAC evaluation criteria, plus benefit-cost efficiency as a sixth 
criterion (this is similar to the structure from a similar origin in Chapter 2, 
Table 2.1). Obviously, since society’s resources are limited, it is crucial that 
they are spent on projects that yield the highest value for money. Benefit-cost 
efficiency in a narrower sense is covered by the ‘efficiency’ criterion of the 
OECD-DAC model, as the relationship between inputs and outputs. We 
found it useful to distinguish between the two levels of efficiency by treating 
them as separate criteria. For benefit-cost analyses, we followed the standard 
method, as presented by, for example, Boardman et al. (2011). The six crite- 
ria’s relation to the logic model is illustrated below in Figure 7.4. 

Our definitions of the six criteria, and the level of success which they rep- 
resent are presented in Table 7.1. 

The purpose of evaluations is to provide an overall picture of the degree of 
public project success. The format is comprehensive enough for this purpose, 
yet minimalistic. With budget limitations, we cannot be too ambitious re- 
garding the methodological rigour when responding to each criterion. Ex- 
perimental designs are rarely realistic for any of the evaluation criteria, and 
certainly not for the strategic ones. Rather, the choice is on simplified evalu- 
ation designs, economic data collection methods, and triangulation between 
various data sources and methods of analysis, quantitative as well as qualita- 
tive, to ensure validity and reliability. 
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Table 7.1 Definitions of the six evaluation criteria 


Level of Evaluation Definition 
Success Criterion 
Operational Efficiency This concerns project implementation and outputs 


in terms of cost, time and quality, and how 
economically the project organisation has 
converted inputs into outputs 

Tactical Effectiveness This concerns whether the agreed outcome has 
been obtained and to what extent the project has 
contributed to this outcome 

Strategic Other This includes all consequences beyond the agreed 

impacts outcome (i.e. side-effects) that can be attributed 
as the result of the project, positive and negative, 
short-term and long-term, for different 
stakeholders 
Relevance A project is relevant if there is a need for what the 
project delivers. Project relevance is measured 
in relation to national political priorities, but 
also stakeholders’ preferences. It is essential to 
bring conflicts of interest to light as part of the 
evaluation 
Sustainability A project is sustainable if its benefits are likely 

to persist throughout its lifetime. This usually 
requires that the total impacts (financial, 
environmental and social) are acceptable in the 


long run 
Benefit-cost This should be measured in terms of total 
efficiency willingness to pay in relation to cost, or 


secondarily in terms of outcome in relation to 
cost (i.e. cost-effectiveness) 


A main feature is that all evaluations should apply the same format. This 
is to facilitate learning, since it allows us to draw lessons from aggregates of 
evaluations. Not only did we apply the same six criteria in all project evalua- 
tions, but we also aggregated the results by setting a score between 1 and 6 for 
each criterion in all projects. This was to facilitate a comparison of achieved 
success across projects. 


7.6 The evaluated projects 


As mentioned, about 300 major projects had been through the government’s 
quality-at-entry scheme since the year 2000, being primarily roads (56%), 
buildings (12%), railway (8%), ICT (7%) and defence (16%). Since the sub- 
sequent detailed planning and implementation period of such large projects 
is extensive, only 120 projects have been completed so far. Of these, about 
half (60) have been in operation for at least five years, and are thus considered 
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Table 7.2 Key information relating to the sample projects. Sorted according to the 


year of evaluation 


No. Project Name Sector Start End Eval. Evaluator 
(yr) (yr) (yr) 
1 E18 Momarken- Road 2005 2007 2012 Concept 
Sekkelsten 
2 Double track Rail 2001 2005 2012 VTI 
Asker—Sandvika 
3 Skjold class MTB Defence 2003 2013 2012 Scanteam, Concept 
4 Customs control area, Building 2004 2005 2012 SINTEF, Concept 
Svinesund 
5 Lofoten fixed link Road 2003 2007 2014 UiN, 
Nordlandsforskning 
6 E6 Riksgrensen— Road 2002 2004 2014 COWI 
Svingenskogen 
7 Eiksund fixed link Road 2003 2008 2014 Menon 
8 NAV ICT Basis ICT 2006 2010 2014 NIBR 
9 Svalbard Research Building 2003 2005 2014 Concept 
Park 
10 Double track Rail 2005 2009 2015 Oslo Economics, 
Sandnes—Stavanger Atkins 
11 Perform ICT 2008 2012 2015 Menon, Vivento 
12 E16 Klofta-Nybakk Road 2005 2007 2015 Urbanet 
13 Military area. Defence 2002 2012 2015 Prokonsult 
Ostlandet 
14 University College Building 2003 2006 2015 SINTEF, Concept 
campus Halden 
15 Rv 519 Finnfast fixed Road 2005 2007 2015 Menon 
link 
16 New Opera House. Building 2005 2008 2016 HR Prosjekt 
Oslo 
17 Halden Prison Building 2006 2010 2016 Oslo Economics, 
Tyrilistiftelsen, 
Sweco 
18 E6 Svingenskogen— Road 2005 2008 2016 Menon, Concept 
Asgard 
19 E6 Asgard—Halmstad Road 2004 2005 2016 Menon, Concept 
20 Gevingasen railway Rail 2009 2011 2017 Concept. SINTEF 
tunnel 
21 Double track Rail 2009 2011 2017 Concept, SINTEF 
Barkâker-Tonsberg 
22  IFI2 building at Building 2006 2010 2018 Concept, 
University of Oslo Multikonsult 
23 Hardanger bridge Road 2009 2013 2018 Menon 
24 Atlanterhav tunnel Road 2005 2009 2019 TOI, Dovre Group 
25 EFFEKT ICT 2005 2013 2019 Menon 
26 GSM-R ICT 2003 2008 2019 SINTEF, Concept 
27 LOS Logistics project ICT 2008 2017 2019 Oslo Economics 
28 TETRA Emergency ICT 2010 2015 2020 Agenda Kaupang 
network 
29 College campus Building 2010 2014 2020 Concept 


Bergen 
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ready for evaluation. So far, therefore, we have evaluated half of all evaluable 
projects. 

These are included in this meta-evaluation: ten road projects, seven build- 
ings, four railway projects, six ICT projects and two defence projects. The 
projects were chosen in chronological order and constitute a relatively repre- 
sentative picture of quality-assured projects in their operational phase (50%). 
Table 7.2 provides an overview, including which evaluators were involved. 
They represent consultancies in Norway and Sweden, and researchers from 
the Concept Research Programme, all of whom were considered independ- 
ent of the projects and the implementing agencies. 


7.7 The evaluation process 


The six criteria framework is not a guarantee of high-quality evaluations in 
itself. Obviously, evaluation skills, impartiality, appropriate data collection 
and analysis methods, etc. are required. 

Each evaluation followed a defined process, which consisted of six steps, 
based on Samset’s (2003) project evaluation textbook and aligned with Mi- 
chael Scriven’s Key Evaluation Checklist (2015). 


1 The Concept Research Programme selected the project to be evaluated, 
and sought acceptance from the responsible ministry (e.g. the Ministry 
of Transportation in the case of road projects). The ministries could, in 
principle, decline, but none of them did. A contact person in the ministry 
(and its subordinate agency when relevant) was identified. 

2 The evaluation team was established, usually following a public call. 
Concept researchers participated in some evaluation teams, in order to 
gather experience in the use of the model. The team consisted of three or 
four people, all with the necessary evaluation skills and knowledge of the 
sector. The scope was set to approximately three people-months of work 
per evaluation, depending on the project’s complexity and availability of 
data. 

3 The evaluation team reviewed and, if necessary, adjusted the project’s 
internal logic. It then operationalised each of the six criteria by selecting 
more specific indicators or evaluation questions to be answered. 

4 The evaluation was carried out by collecting and analysing data, and 
answering the evaluation questions by combining different data sources 
and methods. We leaned on a number of authors who have suggested 
the mixing of methods to improve rigour in the evaluations of complex 
interventions (see for example Green et al. 2015; Ton 2012; and Yin 
2013), and the use of so-called rapid evaluation methods when faced 
with restricted budgets and timelines (Bamberger et al. 2004; Samset 
2003; World Bank 2004). The use of existing literature and expert state- 
ments to assess the integrity of the programme theory was used as a 
supplementary approach to strengthen the validity of findings. Detecting 
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impacts beyond the intended effects normally requires a wide, inductive 
and multidisciplinary approach. 

5 In Step 5, the evaluation team summarised its assessment for each crite- 
rion by setting a score between 1 and 6, where 1 was failure and 6 was 
highly successful. A score of 4 was awarded when the result for the rel- 
evant criterion was acceptable, but not an over-achievement. An overall 
guideline for score-setting was prepared in advance to assist the evalua- 
tion teams. 

6 The final report was made public and distributed. The report and key 
results were stored in a database that is openly available to the public 
(www.ntnu.edu/concept/evaluation-reports). The ministry and respon- 
sible agency were encouraged to follow up the results internally. 


7.8 A meta-evaluation 


In the remainder of this chapter we present some findings and recommen- 
dations based on experiences from the 29 evaluations conducted so far. We 
refer to this as a meta-evaluation. It is based on an update and extension of 
the findings presented in Volden (2018a) based on 20 evaluations. In addition, 
we briefly present findings from a recent survey of the use and benefits of the 
evaluations (Bukkestein et al. 2020). 

The term ‘meta-evaluation’ is ambiguous. Generally, it implies that the 
original analyses of data become objects of a new analysis on a higher level 
(Glass 1976). The much-used UK Magenta Book (HM Treasury 2011) pri- 
marily refers to meta-evaluation as a synthesis of several related evaluations, 
with the purpose of providing some estimate of the average or combined 
effect. This interpretation is close to what Yin (2013) defines as cross-case 
synthesis. On the other hand, Scriven (2015) refers to meta-evaluation as an 
evaluation of one or more evaluations in order to identify their strengths, 
limitations and other uses, against a set of quality standards. A similar in- 
terpretation is suggested by Stufflebeam (2010) who distinguishes between 
three groups of standards: technical adequacy, utility and cost-effectiveness. 

In line with the OECD (2002), we chose to include both the above- 
mentioned interpretations: meta-evaluations are here defined as “evaluations 
designed to aggregate findings from a series of evaluations. It can also be used 
to denote the evaluation of an evaluation to judge its quality and/or assess the 
performance of the evaluators” (OECD 2002, p. 26, our underlining). 

First, in Sections 7.9-7.13, we present an aggregation and synthesis of the 
findings from the 29 separate evaluations, to establish the success of Nor- 
wegian investment projects. The main data source for this part was the 29 
evaluation reports. We coded and summarised the assessments done by the 
evaluation teams, based on a set of questions prepared for the study. This 
included, for each evaluation criterion, the overall score as well as a range of 
more detailed indicators (e.g. efficiency, time, cost and quality, respectively). 
Different measurement scales were used for different aspects, including inter 
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alia the number scale used for score-setting, binary variables (achieved/not 
achieved, etc.) and qualitative descriptions. Accordingly, the aggregation of 
findings across projects was partly quantitative and partly qualitative. The 
scores awarded on each criterion were particularly useful, since they allowed 
us to compute averages and see how they differed across sectors (although the 
number of projects was too small for statistical testing). 

Thereafter, in Section 7.14, we summarise key challenges and weaknesses 
related to the evaluations themselves. This is based on a thorough registra- 
tion and assessment of various quality aspects in the first 20 evaluations, as 
presented and discussed in Volden (2018a). We looked at, inter alia, the inter- 
pretation and application of the six criteria, rigour in design, quality of data, 
triangulation, the use of score-setting, and characteristics of the evaluation 
teams (competence, multi-disciplinarity, impartiality, etc.). Our assessments 
were unavoidably subjective, but meticulously explained and documented. 
We also held a focus group meeting with experienced evaluators who repre- 
sented half of the firms or research institutes that had participated in the Con- 
cept evaluations (cf. Table 7.2). The participants were given the opportunity 
to comment on and share their experiences and assessments of the evaluation 
model, the process, and the need for guidance. The review based on the first 
20 evaluations resulted in an update and extension of the guidelines for eval- 
uators. This led to some improvements in the last nine evaluations, but our 
general impression is that the identified challenges and weaknesses are still 
present to a greater or lesser extent. 

As noted by Scriven (2015), the use or non-use of an evaluation’s results 
cannot be regarded as a quality criterion of the evaluation itself. Admittedly, 
many aspects of an evaluation will affect the probability that the results will 
be used, such as quality and credibility, clear communication, right timing 
and user involvement. But just as often, the failure to use the results and rec- 
ommendations is due to bad management (Scriven 2015). A separate survey 
on the use of evaluations was conducted recently and presented in a Norwe- 
gian report (Bukkestein et al. 2020). In Section 7.15 we briefly present some 
of those findings and discuss what can be done to increase ministries’ and 
agencies’ actual use of evaluation results in the planning of new projects. 


7.9 Findings 


7.9.1 Project success 


Table 7.3 presents the scoring results from all 29 evaluations. Scores 1-2 are 
shown as white, scores 3-4 as light grey and 5-6 as dark grey. 

The overall picture of performance and achievements 1s quite positive, with 
average and median scores between 4 and 5 for all criteria. The weakest result 
concerns benefit-cost efficiency. Evaluators noted that this was the most chal- 
lenging criterion to assess. Generally, there were few really poor scores except 
for one project, that is, the development of an ICT system in the defence 
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Table 7.3 Evaluation results, N = 29 


Project Sector Efficiency Effectiveness Other Relevance Sustainability Benefit- 


Impacts Cost 
Efficiency 
1 Road 3 5 4 5 5 6 
2 Railway 5 3 4 5) 5 3 
3 Defence 4 4 4 5 2 2 
4 Building 6 4 4 5 5 4 
5 Road 4 5 4 3 4 3 
6 Road 5 6 5 5) 5 6 
7 Road 5 6 5 4 5 5 
8 ICT 5 6 4 3 5 3 
9 Building 6 4 5 5 5 3 
10 Railway 4 4 4 5) 5 3 
11 ICT 5 5 5 5 6 5 
12 Road 4 5 3 5 4 4 
13 Defence 5 5) 5 6 5 5 
14 Building 5 4 3 4 5 4 
15 Road 5 5) 5 5 5 6 
16 Building 5 5 6 5 4 4 
17 Building 5 4 5 4 5 4 
18 Road + 5 4 5 4 6 
19 Road 5 5 4 5 4 6 
20 Railway + 3 4 4 4 2 
21 Railway + 3 3 3 4 2 
22 Building 2 5 5 5) 3 A 
23 Road 4 4 3 3 3 3 
24 Road 5 5 5 5 5 6 
25 ICT + 4 4 5 5 6 
26 ICT 5 4 3 5 4 4 
27 ICT 1 2 2 3 2 1 
28 ICT + 4 5 5 5 3 
29 Building 5 4 5 4 6 5 
Average Score 4.3 4.4 4.3 4.5 4.5 4.1 
Median Score 5 4 4 5 4 4 
No of projects that 25 25 23 23 25 18 


scored 4 or better 
No of projects that 15 14 12 18 17 


scored 5 or better 


=. 
=y 


sector (no. 27), which can be considered a failure. Four projects received 
high scores across the board. However, the majority showed more ambiguous 
results, being successful according to some perspectives and less successful in 
others. 

Table 7.4 shows correlations between scores obtained on different criteria. 
It is not surprising that the four strategic criteria are correlated, since they 
concern the same level of success. But we also found correlation across an- 
alytic levels, indicating that a project that is well planned for success in one 
perspective has a greater chance to succeed in other perspectives as well. How- 
ever, score setting is partly subjective, and there is a risk that evaluators make 
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Table 7.4 Correlation matrix (N = 29) 


Correlations Efficiency Effectiveness Other Relevance Sustainability Benefit- 
Impacts Cost 
Efficiency 
Efficiency 1.00 0.28 0.36 0.25 0.59 0.25 
Effectiveness 1.00 0.52 0.29 0.33 0.65 
Other impacts 1.00 0.47 0.50 0.45 
Relevance 1.00 0.47 0.63 
Sustainability 1.00 0.53 
Benefit/cost eff. 1.00 


Table 7.5 Evaluation results: median per sector (N = 29) 


Sector Efficiency Effectiveness Other Relevance Sustainability Benefit- Cost 


Impacts Efficiency 
All 5 + 4 5) 4 4 
Building 5 4 5 5 5 4 
Defence 4.5 4.5 4.5 4.5 35 3.8) 
ICT 4.5 4 4 5 5 3.5 
Railway 4 3 4 4 4,5 2.5 
Road 4 5 4 5 4,5 6 


different assessments, in the same way that some professors are ‘strict graders’, 
while others give out A’s all the time (see further discussion in Section 7.14) 
Although the number of evaluated projects was low, the results indicate 
some interesting sectoral differences, as shown in Table 7.5. Building projects 
scored very well on operational success, but slightly lower tactically and stra- 
tegically. Some of them had ambitious goals that would require more than 
physical facilities to achieve. Railway projects were closely aligned with gov- 
ernment strategies for a ‘green shift’ in transport, and thus were considered 
relevant and sustainable. However, they scored very low on benefit-cost ef- 
ficiency and on effectiveness. The road projects scored high on most criteria, 
but somewhat lower on efficiency, and also on other impacts due to negative 
(environmental) side-effects that had not been properly dealt with. 


7.10 Operational project success 


Efficiency concerns a project’s operational success. The evaluations show that 
25 out of 29 projects scored 4 or better, and 15 scored 5 or better according 
to this criterion. One project stands out with a score of 1. It had an 80% cost 
overrun and considerable delays. 

As shown in Figure 7.5, 21 projects (72%) were completed with a final cost 
below the approved cost-frame. Ideally 85% of projects should keep within 
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the cost-frame, which is normally set at probability level P85. The projects 
also have a target cost which is normally set at probability level P50. The 
evaluations show that 15 out of 29 projects (52%) kept within the target cost, 
which is about as expected. All 29 projects underwent external quality assur- 
ance of cost-frames and steering documents, which certainly seems to have 
been helpful in ensuring realistic budgets. The promising results relating to 
cost control are confirmed in a study including 78 completed projects that 
had been through external quality assurance (Welde 2017). 

No single cause can explain the observed cost overrun in some projects 
in our sample. Explanations are rather project-specific, the three most fre- 
quently mentioned being: 


e Geology/ground conditions (ground investigations not carried out in 
advance) 

e Market situation (tight market resulting in highly priced bids) 

e Additional work not specified in the contract. Unit-price contracts are 
normally used in Norwegian public projects, where the project owner 
carries all the risk. 


There are indications that projects with a shorter implementation period have 
somewhat better cost control than long-lasting projects. Similarly, it seems 
that earlier projects (i.e. those completed in the early 2000s) performed better 
than more recent projects, which is possibly related to economic cycles, or 
perhaps because the best projects were completed first. Project size did not 
seem to affect cost control. 

It should be noted that some of the deviations were considerable, in both 
positive and negative directions. Some evaluators indicated that overruns 


Deviance from cost frame (%), 29 projects 
110% 
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Figure 7.5 Deviance between final cost and cost frame, sorted by size of deviance, 
N = 29. 
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may be explained by the cost-frame being too tight, or conversely that the 
cost-frame might have been too generous in the case of some underruns. 
This suggests that the quality-at-entry scheme may not serve as a guarantee 
for realistic budgets. 

The evaluations found that most of the projects performed well not only 
in terms of cost but also (even more so) in terms of time and quality. 26 out 
of 29 projects (90%) were completed within their time-frame, and 28 (97%) 
were considered acceptable in terms of quality standards. Most projects were 
also considered well organised and managed. Performance in the three parts 
of this so-called ‘iron triangle’ was clearly correlated, in the sense that pro- 
jects with a high score on cost control also had high performance in terms 
of time and quality, compared with projects with a low or medium score on 
cost control. 

Overall, the building construction sector stands out positively in terms of 
operational success. Six out of seven building projects performed excellently 
in terms of cost, time and quality. The evaluations mentioned that several of 
these buildings had been awarded architectural prizes. Road projects expe- 
rienced the largest cost overruns as well as cost underruns, implying that the 
National Public Roads Administration should make efforts to produce more 
accurate estimates. 


7.11 Tactical project success 


With regard to tactical project success, as measured by effectiveness, 25 projects 
scored 4 or better and 14 scored 5 or better, which means that most of the 
projects’ outcomes were acceptable. Three projects received top scores, that 
is, two road projects that realised considerable time-saving and reductions 
in accident levels, and one ICT project that generated improved quality of 
services for users. That said, most of the projects had some room for improve- 
ment in terms of goal achievement. 

Table 7.6 gives an overview of ‘typical goals’ and their achievement in 
each sector. Some of the projects had very ambitious goals, which may partly 
explain their limited achievements. This includes the building projects, some 
ICT projects and the railway projects. Obviously, when a new prison build- 
ing aims to improve the quality of imprisonment in such a way that the 
likelihood of new crime is reduced, this requires more than the physical 
infrastructure. Ambitious goals should have been given more attention than 
was paid in the sampled projects. In recent years, benefits management (BM) 
activities have been implemented in major ICT projects, and are also becom- 
ing a key topic in the railway sector. We believe there is much to learn across 
sectors about how to succeed with benefits management. 

On the other hand, most of the road projects had less ambitious goals 
related to time-saving and traffic safety, their achievement depending 
more directly on the quality of the road itself and requiring no additional 
measures. 
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Table 7.6 Typical goals and their achievement, per sector, N = 29 


Type of Project Typical Project Goals Tactical Tactical Success — Comment 
Success — 
Median 
Score 
Building e Co-locate 4 Goals only partly achieved. 
(7 projects) departments A Direct effects are observed 
and B and realise (e.g. user satisfaction with 
professional synergies the building), but effects 
e Effective on business achievement 
rehabilitation of is more uncertain and 
prisoners — through often limited. No (or 
improved facilities very little) active benefits 
e Effective research and management is carried out 


education — through 
improved facilities 


Defence e Defence of coastal 4,5 Only two projects, both 
(2 projects) areas (new vessels) with acceptable goal 
e Increased defence achievement, but also 
capability (training difficult to measure 
fields) 
ICT e More efficient 4 Goal achievement varies 
(6 projects) administration of considerably between the 
welfare services six projects, and is difficult 
e Efficiency and to assess. Not all projects 
management within have being carrying out 
defence benefits management 
e More efficient activities 
emergency services 
Railway e Travel time-saving 3 Limited effects so far, but 
(4 projects) + Sustainable transport may potentially increase 
through modal if/when other measures 
shift (improved rail are implemented (co- 
competitiveness) ordination of timetables, 


restrictions of car use, 
etc.). Benefits management 
activities only introduced 


recently 
Road e Travel time-saving 5 Travel time-saving and safety 
(10 projects) through shortening goals are largely achieved. 

of a road or increased In some projects wider 
capacity benefits for the economy 

° Traffic safety were also mentioned as 

e More efficient goals, but such effects 
transport between could not be seen 


island and mainland 
(bridge/sub-sea 


tunnel) 
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7.12 Strategic project success 


The other four evaluation criteria express strategic project success. The eval- 
uators concluded that the projects performed acceptably in these dimensions 
too. All projects but two scored 4 or better on at least one strategic criterion. 


7.12.1 Other impacts 


As many as 23 projects scored 4 or better on other impacts, and 12 scored 5 
or better. Several projects generated positive side-effects, such as city devel- 
opment, learning effects (the project being a pilot for new working methods, 
etc.) and additional user groups. One project, the New Opera House in Oslo, 
received top score for its positive effect on subsequent urban development. 
Generally, the building projects did best in terms of maximising positive 
side-effects, typically depending on the location of the building. Some pro- 
jects could have done more to avoid negative side-effects, in particular some 
road and railway projects with negative effects on nature and the local en- 
vironment. The majority of projects, however, had no extensive positive or 
negative side-effects. 


7.12.2 Relevance 


Twenty three projects scored 4 or better on relevance, 18 scored 5 or better, 
and none scored below 3. This implies that all were considered solutions 
to real problems. Some ICT projects were ‘must-haves’ to facilitate a re- 
form, etc. 

However, the relevance criterion is in itself often multidimensional, and 
some projects involved conflicts of interest (i.e. they were not equally rele- 
vant in all perspectives). This was the case with regard to the Lofoten fixed 
link, a road project that connected a remote region to more urban areas, but 
left the neighbouring remote region even more isolated. 

In some cases, evaluators suspected that other, more relevant solutions to 
the problem existed (at least more cost-efficient ones). For example, some of 
the studied ICT projects involved the development of new, complex solu- 
tions, although simpler, off-the-shelf systems already existed. Similarly, bet- 
ter route choices existed for some of the railway projects. 


7.12.3 Sustainability 


Twenty five out of 29 projects scored 4 or better on sustainability, and 17 
scored 5 or better. This implies that project benefits were largely expected 
to continue over a sustained number of years. For example, transport facili- 
ties were considered to have sufficient capacity within their planned lifespan 
(40-75 years), and most new buildings had expansion possibilities on site. 
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However, projects had to be sustainable in all aspects (i.e. financial, en- 
vironmental and social) to be assigned a top score. For example, the Opera 
House project was considered to have long-lasting effects on cultural policy 
goals, but was not given a top score because growing operational and main- 
tenance costs made it financially unsustainable. In this case (and others), the 
project’s sustainability depended on increasing public subsidies over time. 


7.12.4 Benefit-cost efficiency 


Overall, the projects scored slightly lower on benefit-cost efficiency. In total, 18 
out of 29 projects scored 4 or better, and 11 scored 5 or better. 

We found some clear sectoral differences. The five most benefit-cost effi- 
cient projects were all road projects in urban areas. On the other hand, all the 
railway projects scored relatively low. It should be noted that evaluators found 
this criterion difficult to assess. Often, benefits could not be monetised and 
had to be assessed qualitatively (for example in defence and building projects), 
resulting in high uncertainty and low comparability across projects. 

Interestingly, the low benefit-cost efficiency for some projects was known 
by decision-makers ex ante. This indicates that the CBA had not been crucial 
to project selection. In fact, some of the road projects that proved to have a 
positive net present value ex post were estimated to be negative ex ante. 


7.13 Discussion 


In most of the cases, the evaluators concluded that the projects were success- 
ful in more than one aspect, especially in operational terms. There appears 
to be some correlation between the scores for the various criteria. This is not 
surprising, since a well-thought-out and carefully planned project will nor- 
mally be successful in several respects. However, there may also be conflicts, 
for example when some of the projects scored high on relevance and sustain- 
ability, and lower on benefit-cost efficiency. 

All four railway projects were well aligned with the government’s strat- 
egy for sustainable transport (although one scored low on relevance due to 
the unsuitable location of a station). However, with passenger numbers less 
than estimated by the planners, and a relatively high capital cost, the value 
for money was considered low. We agree that not all projects can or should 
be ‘profitable’, but one should at least consider whether a simpler solution, 
still with acceptable goal achievement, would substantially improve value for 
money. Similarly, a project with high effectiveness but negative side-effects 
should perhaps be redesigned to achieve a better overall score. 

One noteworthy observation is that many of the projects that scored high 
in tactical and strategic terms were not aimed at specific stakeholder groups 
or regions, but rather followed from national political objectives. This sup- 
ports earlier findings that when specific stakeholder groups manage to mobi- 
lise government funding for ‘their’ project, the project may turn out to be less 
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relevant from the perspective of the wider society — a phenomenon known as 
perverse incentives in project selection (Volden 2018b). 

The positive results found in the sampled projects seem to conflict with the 
public discourse and studies that demonstrate a low level of success in public 
projects. For instance, Flyvbjerg et al. (2009) use the expression “over budget, 
over time, over and over again” and explain the widespread problem of cost 
overruns by delusion and deception. We think that caution should be used 
when referring to public projects as generally unsuccessful. First, the media as 
well as academic literature are largely concerned with cost control, which is 
only one aspect of project success. Second, as noted by Love et al. (2015), dif- 
ferent empirical studies on cost control come to quite different conclusions, 
depending, inter alia, on the point of reference from which a cost overrun is 
measured (those that find the largest overruns typically compare with early 
and uncertain estimates). The 29 Norwegian projects stand out as relatively 
successful, which can be explained, at least to some extent, by the quality- 
at-entry requirements which ensure that they were thoroughly planned and 
reviewed before being submitted to Parliament for approval and funding. It 
should be noted that the remedy suggested by Flyvbjerg et al. (2009) to avoid 
delusion and deception was to take an outside view on project planning and 
estimation, which is exactly what the quality-at-entry scheme does. 

On the other hand, we cannot eliminate the risk that some scores were 
positively biased. A study by Wiig and Holm-Hansen (2014) found that in 
evaluations of development assistance projects, positive side-effects were men- 
tioned more often than negative ones. They suggest that evaluators may be 
reluctant to criticise without hard evidence on which to base their criticisms, 
but willing to mention a positive issue that has the same level of uncertainty. 
While acknowledging the general risk of positive bias, we believe it was 
moderate to low in the 29 evaluations. The main reason is that the evalua- 
tions were organised by a third party, and all the teams were entirely inde- 
pendent of the projects. 

However, another and more pertinent matter is how to ensure that the 
scores were well-calibrated across projects. The scores were set by a different 
evaluation team in each case, based on the team’s subjective assessments of 
findings. As will be discussed in the next section, it is crucial to have clear 
guidelines for score-setting. In particular, the level of ambition inherent in 
the goal hierarchy must be considered when deciding on the score for effec- 
tiveness. We suspect that different levels of ambition relating to project goals 
may explain some of the sectoral differences when it comes to effectiveness. 


7.14 Lessons learned about the evaluation framework and 
procedure 


The evaluations provided useful experience on the evaluation model and our 
own practice. A thorough review of various quality aspects of the first 20 
evaluations led to an update of the guidelines for evaluators, which, in turn, 
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led to some improvements in the last nine evaluations. However, our general 
impression is that the identified challenges and weaknesses were still relevant, 
and ought to be taken into account when conducting evaluations. 


7.14.1 General experiences of the model 


It seems clear that the chosen framework worked well, and that the six cri- 
teria covered the main aspects of public project success. Some experienced 
evaluators noted that the strategic criteria were ‘eye-openers’. Knowing that 
pure economic evaluations are often considered by decision-makers to be too 
narrow (Nyborg 1998), our evaluators agreed that the six-criteria framework 
was more relevant. 

The process of disaggregating the criteria into specific indicators and then 
aggregating the findings to provide answers to each criterion provides a good 
balance between the need for standardisation and flexibility. The evaluations 
converged more and more into a common form, and their quality improved 
over time. 

However, we also see some challenges related to how ‘other impacts’, ‘rel- 
evance’ and ‘sustainability’ were interpreted. Some evaluators treated these 
strategic aspects superficially, realising that they cannot be measured in quan- 
titative terms. Others interpreted them too narrowly, as found by Chianza 
(2008). Others still confused them with benefit-cost efficiency, downplaying, 
for example, the environmental, social and ethical concerns. The evaluators 
confirmed that they were sometimes uncertain about the interpretation of 
these criteria. 


7.14.2 Methodological rigour to be balanced against available 
resources 


The evaluations comprised six criteria, each of which required proper treat- 
ment. At the same time, the budget and time available implied clear limi- 
tations regarding scope and methodological rigour. Our evaluations were 
undoubtedly ‘rapid’, and the scores sometimes uncertain. This is not uncom- 
mon in evaluations, and must be accepted. However, it is crucial that the 
choice of methods and limitations are communicated. These findings support 
those of Samset (2003), Bamberger et al. (2004) and others who have argued 
that it is possible to conduct evaluations of acceptable quality under budget, 
time and data constraints. 


7.14.3 The need for a broad approach to data collection and analysis 


Different methods of data collection and analysis are needed for different 
evaluation criteria. For efficiency, data from project reports were used, as 
well as interviews and benchmarking of cost data with similar projects. 
For effectiveness, time-series data for outcome indicators were used (often 


Closing the loop 181 


including comparison groups, such as similar geographic regions without the 
new infrastructure) and interviews with a wide range of stakeholders. For 
the strategic criteria, the evaluators normally used a combination of differ- 
ent sources, predominantly qualitative ones. For benefit-cost efficiency, they 
used all existing data and a set of assumptions and price tags. All evaluations 
included site visits. 

It is costly to collect primary data, and evaluators must therefore prioritise 
carefully. Generally, the evaluators for the 29 projects studied chose simple 
and informal methods. 

Most evaluators used triangulation to an acceptable extent, but some fo- 
cused too much on quantitative data and experiment as the gold standard. 
For example, one evaluation report devoted more space to discussing the 
difficulties of quantifying benefit-cost efficiency than to describing it with 
alternative data. The quality of an evaluation with limited time and budget 
rests strongly on the ability to use a broad approach with a wide range of 
sources and methods. 


7.14.4 The project logic ought to be reviewed 


Reference data, in terms of descriptions of the goal hierarchy or logic model, 
often has weaknesses in project evaluations (Samset et al. 2014). The quality- 
at-entry scheme required each project to have a defined goal hierarchy. De- 
spite this, more often than not, there were problems, such as a missing causal 
logic or the wrong level of ambition (too high or too low). Evaluators ought 
to take such weaknesses into account, but unfortunately some were not criti- 
cal enough. Ideally, they should re-establish the logic model as it should be, as a 
basis for their evaluation, in accordance with the project owner. At least they 
should adjust for the level of ambition when setting scores. A project should 
not be awarded a score of 6 if its goal was trivial, and likewise, it should not 
be awarded a score of 1 if the goal was unattainable. 

A theory-based approach to evaluation might also be helpful. Scholarly 
literature can be used to assess whether certain changes are likely to be an 
effect of the project (i.e. the attribution problem). Kaplan and Garrett (2005) 
mention that a common example of theory failure is to assume that a new 
technology or infrastructure will make people change their habits without 
additional measures, such as training and financial incentives. A review of 
the programme theory could have revealed such a failure, and may similarly 
reveal redundant project components. 


7.14.5 Broad and impartial evaluation teams 


It is required that evaluation teams have no relation to the projects they eval- 
uate. Furthermore, that they have expertise within evaluation, economics 
and project management, and some knowledge of the sector and type of 
project. As noted by Scriven (2015), an evaluation team must be broad and 


182 Knut Samset and Gro Holst Volden 


represent different perspectives and disciplines, as this 1s essential for compre- 
hensive and balanced assessments. 

In our evaluations, impartiality was well assured, and we have not yet ex- 
perienced lack of confidence in the evaluations from stakeholders or the wider 
public. On the other hand, some of the teams consisted primarily of econo- 
mists and were thus considered insufficiently broad and multi-disciplinary. 
This may explain why some evaluators interpreted ‘other impacts’ from a 
narrow, economic perspective. 


7.14.6 Score-setting needs to be better calibrated 


Score-setting was an essential part of these evaluations. Experience indicates 
that the use of scores is valuable for drawing lessons across projects and sec- 
tors. However, it is essential that results are well-calibrated. Scores were set 
by different teams who may interpret and use the scale differently. Subjective 
judgements regarding the choice of indicators as well as score-setting cannot 
be avoided, but can be minimised using clear guidelines for evaluators. A 
common problem in the first 20 evaluations was subjective score-setting. 
Later, the guidelines were updated, leaving somewhat less room for discretion. 


7.15 How to ensure that ex post evaluations contribute 
to learning 


The purpose of doing ex post evaluations is, ultimately, to improve the suc- 
cess of future projects. For this to happen, it is not enough that evaluations 
are of good quality. The results and recommendations must also be perceived, 
understood, and used. 

It is well known that evaluations are not always used as intended (Dahler- 
Larsen 2012; Samset 2015). Eliasson et al. (2015) showed that cost-benefit 
analyses of Norwegian road projects that were performed ex ante did not 
have any effect on which projects were selected for implementation. In a re- 
view of state enterprises’ use of evaluation, one of the main findings was the 
lack of follow-up of evaluation results (Agenda Kaupang & Ramboll 2016). 
At worst, evaluation becomes symbolic rather than truth-seeking. Samset 
(2015) illustrates this aptly by describing the results from a study on the use of 
evaluation in the development aid sector. Officials and decision-makers were 
asked to identify which information sources they considered most important 
for learning. Of 19 different sources, evaluation reports were in 15th place, 
and were thus considered among the least important sources of learning, de- 
spite the extensive scope of evaluations in the field of development aid. 

Our evaluations were performed ex post, when it was too late to make 
changes in the project as such. However, the findings should be useful to 
improve appraisal tools and models, to reduce inaccuracies in estimations of 
costs and benefits, and to improve project management tools and procedures 
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to be used in future projects. Thus, what we are discussing is indirect use of 
evaluation results. 

A survey on the use of Concept evaluations by the target group (i.e. min- 
istries and government agencies) was performed recently (Bukkestein et al. 
2020). Interviews were conducted with senior project owners and employees 
involved in project planning and appraisal in affected ministries and agencies. 

A key finding was that the target group had varying knowledge of the 
evaluations concerning their own sector. Those who had been involved and 
were aware of the evaluations often showed considerable interest in them. 
But generally, the evaluations were not sufficiently anchored in responsible 
ministries/agencies and the results were not disseminated sufficiently widely 
after the evaluations were completed. 

Furthermore, very few (hardly any) evaluations were initiated by the min- 
istries and agencies themselves. Project appraisal and planning was thorough 
and systematic, and for the largest public projects, external quality assurance 
was conducted according to the Ministry of Finance’s scheme, but ex post, 
only a few evaluations had been carried out. These evaluations focused on 
operational success and had essentially a control function, rather than learn- 
ing as their main focus. 

Bukkestein et al. (2020) provide a set of recommendations on how to im- 
prove the use and benefits of evaluations, the most important being: 


e The ministry/agency should be involved in the evaluation process, ear- 
lier and more systematically than they are today. They should also be 
involved in discussing the results and their implications. 

e Evaluators should distinguish more clearly between informants/data sup- 
pliers on the one hand, and the target group of the evaluations on the 
other, although the same people often hold both roles. The latter role is 
easily forgotten. This is particularly a problem when evaluations are out- 
sourced to external evaluators, as practised by the Concept programme. 

e When conducting a broad six-criteria evaluation, we need to acknowl- 
edge that different parts of the evaluation may have different target 
groups. Normally, operational project success concerns the agency most, 
tactical success concerns both agency and ministry, and strategic success 
concerns the ministry most. 

e One may consider separating the three levels of success into three sep- 
arate evaluation parts, and even carrying them out at different times. 
Instead of waiting five years until the project can be evaluated in all three 
perspectives, its ‘efficiency’ can be evaluated much earlier, when the re- 
sults will be more relevant for the executing agency. 

e The target group is less interested in the score they obtained on a cer- 
tain project from the past, and more on how to improve their next pro- 
ject. Some of the earliest evaluations were only concerned with finding 
the right level of project success, and had less focus on the explanations 
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behind good or bad results. More recently, therefore, a separate chapter 
of ‘learning points’ is included in the evaluation reports. 

e Evaluation results must be easily available and communicated. Much is 
done already to disseminate the results. Short and informative summaries 
are presented on the Concept website, and oral presentations are offered 
to the ministry and agency as the report is launched. This dissemination 
work must be continued and strengthened with help from the organisa- 
tions themselves. 


7.16 Conclusions 


Ex ante evaluation is needed to link the decision processes to the results of 
overriding reasoning based on facts and systematic analysis. The result of a 
decision-making process is not necessarily in accordance with this require- 
ment. The final choice can be better or worse than that recommended at the 
outset. However, in any case, the best-case result comes from democratic 
processes that legitimise the outcome. The criteria for ex ante evaluation 
are often associated with a rational ideal of strong management and clear 
thinking. 

Ex ante evaluations should also build on experience from ex post evalua- 
tions, preferably in sufficient numbers to provide a greater scope of experi- 
ence, and should at the very least include several alternative courses of action. 
Moreover, the evaluation criteria in ex ante evaluation ought to correspond 
to a degree with those of ex post evaluation. This is particularly important 
when constructing learning loops that function over time. Not least, it is 
important that ex ante evaluation is made to a central reference in the subse- 
quent decision-making process against which the various alternative courses 
of action are evaluated. 

What can be learned from such an analysis? First, the decision-making 
process associated with major public investment projects is complex, which 
implies that learning should be complex and versatile. It is important to be 
aware of various considerations and difficult trade-offs. Hence, our analysis 
functions as an ex post evaluation. At the same time, the absence of clear or- 
ganisational thought and rational calculation, as a reflection upon the central 
aspects in the early phase of a project’s life, affords considerable potential for 
ex ante evaluation. Second, ex ante evaluation and its utilisation is not an ob- 
jective or unambiguous process, but rather a political and administrative pro- 
cess. Various actors, such as political and administrative leaders, may use these 
evaluations in differing ways. Some may interpret them in their own interests 
and accordingly slant undertakings and learning, whereas others may adhere 
more closely to conclusions and recommendations suggested by the evalua- 
tors. In today’s societies, it is relatively common that either evaluations are 
clearly managed, or that results are interpreted, slanted, politicised and over- 
sold. This analysis may contribute to more control and greater breakthrough, 
but may also undermine the credibility and support of projects. 
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In Norway, ministries and agencies with large investment projects have 
become quite good at ex ante evaluation (appraisal) and planning. Since the 
year 2000, project decision documents have gone through external qual- 
ity assurance. As discussed in Volden and Samset (2017), this has provided 
decision-makers with a better basis for selecting which projects to implement. 
The scheme provides a more systematic approach to the early identification 
of project ideas than the prior system. Rather than going straight to select- 
ing road sections and determining a technical solution, planners are forced 
to take a broader perspective and to discuss societal aspects, which allows 
ideas to mature and stimulates creativity in the agencies. Thus, the scheme 
has done much to remove or curb several of the paradoxes described in the 
‘Paradoxes’ paper. 

The assumption is that this will also contribute to improved project per- 
formance. However, ex post evaluation is not part of the scheme, which is a 
paradox in itself. In general, ex post evaluations of government investment 
projects are still rare. Worsley (2014) referred to ex post evaluations as “the 
weak link” in the assessment process for transport projects in OECD coun- 
tries. This is perhaps not surprising. In contrast to, for example, health or 
educational programmes, an infrastructure project cannot be implemented 
stepwise. Therefore, it could be argued that whereas good planning is cru- 
cial, ex post evaluation is a waste of time and resources. However, that would 
be an erroneous conclusion because there is much to learn from one project 
to another, both within and between sectors. Given the poor reputation of 
public projects in high-income countries in general (Flyvbjerg et al. 2003), 
the potential to improve project practices is considerable; so is the potential to 
improve project planning, governance and the quality at-entry scheme itself. 

Evaluation should be based on the project’s logic model, as discussed above. 
It should ask not only about economic aspects but take a broad and multifac- 
eted view on project success. In their most recent economic survey of Nor- 
way, the OECD (2017), focusing primarily on transport projects, suggested 
that ex post evaluation of projects should be conducted more systematically, 
and that a broad framework should be applied, to strengthen scope, accuracy 
and credibility. 

Systematic ex post evaluation of the largest public projects is now being 
implemented by a research programme, with funding from the Ministry of 
Finance, as illustrated above. We applied a generic evaluation framework 
inspired by the one recommended by the OECD-DAC for the evaluation of 
development assistance projects and programmes. We described the develop- 
ment of an evaluation practice that started in 2012 by testing the six-criteria 
framework on four different projects. The experiences with the model were 
good, so evaluations continued, and over time we developed better guide- 
lines for score-setting and other issues. After 29 evaluations, we can begin 
to draw conclusions about the success of the projects. Recently, we have also 
begun to discuss what more can be done to ensure that the evaluation results 
are used and contribute to learning. Over time, we will build a large and 
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hopefully useful database with results from completed projects, all evaluated 
according to the same framework. 

A key finding so far is that most of the projects were successful in one or 
more perspectives, as considered by the evaluators. This contrasts with the 
public discourse and studies by Flyvbjerg and others which conclude that 
public projects are unsuccessful. Some projects scored high on relevance and 
sustainability, but low on benefit-cost efficiency, and vice versa. This type 
of deviance needs to be communicated to project owners and various stake- 
holders, who might have conflicting views on the weighting of the criteria. 
The evaluations thus provide a basis for discussing whether a better balance 
between different concerns could have been possible. The possibility to com- 
pare and learn across different sectors is also considered useful. Some sectors 
are better at cost control, others at benefits realisation, still others at sustain- 
ability, etc. 

The findings of the evaluations should be useful for the purpose of account- 
ability, as well as for learning and improvement. Although each evaluation 
was limited in time and resources, it may identify major risks and problems 
that should be examined in more detail by the responsible ministry. Further- 
more, the findings should provide input to the appraisal and planning pro- 
cesses of future investment projects funded by national government. It should 
be noted again that the sample of projects was not statistically significant; 
thus, any attempts to generalise findings should be regarded as preliminary 
and tentative. Over time, when the database of evaluated projects is larger, 
it should be examined as to whether the patterns described above still hold. 


e We argue that ex post evaluation according to our suggested model will 
be helpful in overcoming all ten paradoxes from the ‘Paradoxes’ paper to 
a greater or lesser extent: 

e The six-criteria framework implies a broader perspective on project suc- 
cess than is normally taken. Hopefully this will inspire planners to focus 
more on strategic issues as a basis for project selection (cf. Paradox 1, ‘the 
success paradox’ and 2, ‘the significance of front-end management’). 

e The simple framework, if used ex ante, with a special focus on ‘relevance’ 
and ‘sustainability’ early on, should also help planners single out impor- 
tant information and avoid information overflow (ref. Paradox 3, ‘the 
paradox of early information overflow’). 

e This, together with the building of a database that includes a wide range 
of solutions to various problems may also open planners’ eyes to the op- 
portunity space that exists on when to solve their next problem (cf. Par- 
adox 4 ‘the paradox of the opportunity space’). 

e It is essential that a projects goal hierarchy constitutes a logic model 
with goals at different levels being connected. Ex post evaluation may 
demonstrate common weaknesses in goal structures, which may, in turn, 
be used to improve strategic alignment in future projects (cf. Paradox 5, 
‘the paradox of strategic alignment’). 
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e The building of a database with actual results from completed projects 
may also curb the problem of early cost estimates being unrealistic (cf. 
Paradox 6, ‘the cost estimation paradox’). The database may thus facil- 
itate a system often referred to as Reference Class Forecasting, that is, a 
way of predicting the future by looking at similar past situations and their 
outcomes. 

e A paradox that should definitely be affected by ex post evaluation is 
Paradox 7, on disregarded analyses of costs and benefits. When estimates 
are well-documented and based on results from past projects rather than 
pure guesswork (as is often the case today), it will be more difficult for 
decision-makers to ignore them. 

e Evaluation may also curb the ‘predict & provide’ paradox (Paradox 8), 
especially if it makes planners focus more on the ‘relevance’ criterion. 
Assessing a project’s relevance implies taking a step back and asking what 
the problem is, not simply assuming that a constant expansion of the in- 
frastructure in line with some trend is the best solution. 

e The use of actual results from completed projects may also prevent per- 
verse incentives from materialising, at least in the form of systematically 
skewed benefits and cost estimates (cf. Paradox 9, ‘the paradox of per- 
verse incentives’). 

e Finally, by including the ‘sustainability’ criterion as part of the evalua- 
tion, planners will widen their horizons and assess long-term benefits 
and costs, also ex ante (cf. Paradox 10, ‘the paradox of myopic decisions’). 


The evaluators’ experiences of the evaluation framework were largely posi- 
tive. They found it relevant and meaningful that evaluation was not limited to 
aspects of project management success, which has traditionally been the focus 
in the project management community. Neither was the framework limited 
to benefit-cost efficiency, which is normally the focus in the transport sec- 
tor. (Other sectors rarely conduct evaluations at all). Instead, the six criteria 
covered intended and unintended effects alike, together with goal-oriented 
and efficiency perspectives, and explicitly raised questions about long-term 
sustainability. Also, this meta-evaluation revealed some improvement points, 
from which the lessons learned are continuously used to improve the frame- 
work, processes and guidelines for future evaluations, how the evaluation 
teams should be put together, how the criteria should be understood, and 
clear, common principles for score-setting. 

Ministries and government agencies need to see the benefits of the evalu- 
ations and their learning potential. It is still too early to determine whether 
these 29 evaluations will lead to improved practices, but so far it seems that 
they have not been used to the extent that we had hoped for. This is hardly 
surprising, as the non-use of evaluations has been a topic within the evalua- 
tion literature since the 1970s, and it is well-documented that many evalua- 
tions are more ritual than reality. We believe there should be a system where 
planners are obliged to request, and use ex post results from previous projects 


188 Knut Samset and Gro Holst Volden 


when planning a new project. That is, the use of evaluations should be sys- 
tematically integrated in organisations’ project governance frameworks. 

Over time, hopefully, many project evaluations will be produced corre- 
sponding to this framework. One ambition is to further improve their quality 
and ensure that scoring becomes better calibrated over time. Since the pro- 
jects in each sector have similar outcomes, allowing for standardised meas- 
ures, the resulting evaluation database would then provide a valuable basis for 
robust practices and better determinants of government investment projects’ 
success. 
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8 Conclusions 
Terry M. Williams 


8.1 The setting of major public projects 


This book has been discussing the genesis, planning and delivery of ma- 
jor public projects. Projects are the key organisational form used to deliver 
transformational policy change and build new systems in the public sector. 
This includes not just obvious physical systems, such as military systems 
and infrastructure construction, but also public change and transformation 
programmes, major IT systems, and programmes aimed to achieve major 
policy aims, such as decarbonising transport and energy. As a leading offi- 
cial in delivering such programmes in the UK stated, “The vast majority of 
government policies are delivered through projects of various forms” (Meggs 
2017, p. 3). 


Currently, more than 20% of global economic activity takes place as 
projects, and in some emerging economies it exceeds 30%. World Bank 
(2009) data indicate that 22% of the world’s $48 trillion gross domes- 
tic product (GDP) is gross capital formation, which is almost entirely 
project-based. In India it is 34%, and in China it is 45% [of GDP]. 
(Scranton 2014, p. 1) 


We therefore have considerable experience of undertaking such projects. 
However, it is well-known that these efforts have not always been viewed as 
successful, either by the public or by governments. Samset and Volden (2016) 
looked at data in Norway on how such projects were set up, and found a 
number of ‘paradoxes’ or dissonances which need explaining so that we can 
understand how to avoid or overcome them. This book has been considering 
those paradoxes. 

So what are the ‘paradoxes’ that are the subject of Samset and Volden’s 
paper? Are they true paradoxes in that two contradictory positions appear to 
be true (such as the exploration/exploitation contrast in organisational ambi- 
dexterity)? Or are they simply examples of the phenomenon frequently seen 
in organisations of the ‘Knowing-Doing gap’ (Pfeffer & Sutton 1999)? Do we 
really know how to make our projects deliver, and if we utilised the lessons 
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of Pfeffer and Sutton, could we develop our organisations so that they more 
often deliver projects in what we know to be the better way? 

It appears from the last six chapters that the aforementioned paradoxes are 
more subtle than either of these, and are inherent to some extent in the front- 
end of any project, the evolutionary process of which is embedded within its 
environment. Such paradoxes are exacerbated by the nature of major projects 
in the public and political domain. They naturally occur in actual practice, 
and we can recognise them, the suggestion being that these paradoxes give 
rise to behaviours that lead to inadequate analysis and decision-making, and 
disbenefits to the final outcome. The object of this book has been to explore 
such paradoxes, to explain why they arise, consider how to manage them, and 
where possible, to suggest what behaviours we should be encouraging to try 
to avoid their negative effects. 

How have we approached these questions? Our approach has been to look 
at projects as they are embedded in actual public practice. There has been a 
lot of research in the area of projects and project management in the public 
domain, mainly falling into one of two types, and leaving a gap in what we 
know: 


i There is considerable research on what organisations need to do and why, 
which regards projects as individual ‘black box’ entities: strategy, societal 
transformation, economic geography, etc. — well-grounded but treating 
projects as entities that realise strategy. There is also a stream of research 
looking into what projects are set up to achieve and whether or not they 
achieve this: key here is a stream of research by Flyvbjerg and colleagues — 
most famously in Flyvbjerg (2003), but with considerable later research 
(although there is also a stream of research arguing against some of their 
findings and pointing in particular to Hirschman’s ‘Hiding Hand’ idea, 
e.g. Ika et al. 2021). 

ii There is also considerable research on ‘project management’, looking 
within a project, assuming a specific task has been well-defined and 
presented for a project team to achieve, and ignoring the environment 
around the project. These were traditionally rather theory-light, norma- 
tive studies, but recent years have enlarged the scope of such work. 


e Neither looking solely at the environment or solely at the project is 
sufficient. The move recently into ‘Project Studies’ (e.g. Geraldi & 
Soderlund 2016) recognises the need to study both the project itself 
and its environment — and the complex interfaces between them (some 
of the research by Ika has also started to look ‘under the bonnet’ of 
projects). A ‘project’ is not a separate entity from its environment, but 
we must understand how both work together, particularly as the pro- 
ject is formulated and planned. Hence we have taken such a holistic 
approach in this book. 
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e The book has tried to explain behaviour in the setting-up of projects 
within their real environments. Each chapter has taken a different 
aspect of the front-end of the project, offering a depth of material 
that warrants careful study and provides lessons for the practising 
manager. This chapter takes a cross-chapter view, and brings together 
some thoughts on seven ideas that crop up in most, if not all of the 
chapters: the problem or need that triggered the problem; the jump to 
an early project solution; stakeholders and consultation; information 
generation and flow in the project; accountability for the results of the 
project; and reflecting back on a project. 


8.2 The front-end as a process 


Traditional thinking, as described above, has looked at the project as an ex- 
ercise to carry out a defined task in a specific time, at a specific cost. As we 
have looked at the paradoxes, we have seen that these arise because the de- 
cision to carry out a project, and the definition of what that project 1s, is not 
a specific point in time, but a process that can take considerable time. The 
generally accepted discussions and analyses of such activities often see front- 
end processes as being outside of time, based on the implicit assumption that 
their own dynamics should not impact the results of the project, but this is 
clearly not correct. We have been exploring this process that derives from the 
initial idea or recognition that ‘something must be done’, to the point where 
a project is agreed, ‘signed off’ and made to start — what is generally known 
as the ‘front-end’ (Williams et al. 2019). 

This process might not be straightforward, even in simple projects within 
a private company. Within the public sector, the project has to emerge from 
a mass of stakeholder interests and political influences. There is a need to 
communicate between all the different factions, and Chapter 4 takes the 
idea of representation construction to conceptualise the project front-end — 
not only internal representations but also external representations, which 
are key for communicating about the project and eliciting assessments from 
decision-makers and stakeholders. This takes place over time, with changes 
and revelations in the process, as discussed below. Even then, the various 
stakeholders and decision-makers do not share common goals, and, indeed, 
might be driven by a variety of cognitive, emotional and social reactions, so 
that decision-makers will also need to be in a process of filtering and balanc- 
ing those views. 

This process cannot be seen as a sequence of rational, logical decisions, but 
must take into account the above influences. We have been looking at how it 
explains the paradoxes, and how we can use this knowledge to design better 
front-ends. This has been the driver for the chapters. Chapter 2 considered 
how project aims are defined; Chapter 3 looked at the logic behind the pro- 
cess, and in greater depth at how one major public body’s front-end process 
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aimed to tackle these issues; Chapter 4 used the processual nature of the 
front-end to explain the trade-offs that must necessarily be made. 

Perhaps the clearest illustration of the temporal effect of the front-end is 
the activity of project estimation. While well-researched as a ‘one-off’ activ- 
ity, this book has been exploring the effects of the processual nature of the 
front-end. Chapter 5 traced the changes to estimates as information became 
available in the front-end, quoting a parliamentary committee report stat- 
ing that ministers were too keen to commit to cost and timescale estimates 
(which are usually stated deterministically, i.e. with no uncertainty bounds, 
despite the early stage of development) early in the process. They go on to say 
that “project managers become tied to these estimates. The early estimates 
can shape the rest of the project delivery ...”. The earliness of these estimates 
can be forgotten later, as only the final, approved estimate is remembered 
(the ‘cost estimation paradox’), but early estimates can shape a project if they 
are wrongly relied upon. The extensive case study presented in Chapter 6, 
exploring many of the paradoxes illustrates this. 

These chapters, of course, overlap, and we have tried to include cross- 
references between them. The ‘paradoxes’ themselves are not ten independ- 
ent paradoxes, but are related causally. This is a simplification, many of the 
paradoxes being mentioned in the various chapters, but Figure 8.1 shows 
where the interests of the chapters are most focused, and how the sequence of 
chapters to some extent follows the causal effects of the paradoxes. 


8.3 The project trigger 


As can be seen, we should first consider what has prompted the project (Chap- 
ter 3 uses the term ‘project trigger’). What are we actually trying to achieve 
in our project? Success is multifaceted, but primacy should be given to the 
objective for embarking on a project. If we concentrate purely on the tactical 
objectives of the project (time/cost/scope), we are missing the point. The part 
of government which wants (or needs) to achieve something ought to con- 
sider first what it is trying to achieve and why, in the short and long term — 
including the long-term effects on sustainability (as discussed in Chapter 4). 
It is only then that we can logically move on to the ‘front-end’ of the project. 

These are easy statements to make, and Chapter 2 explored the difficulty 
and complexity of establishing a strategic project aim in the public sector. 
Such complexity includes the wide range of heterogenous stakeholders with 
different aims, the frequent difficulty of specifying, let alone quantifying tar- 
gets for the ‘public good’, the added difficulty of comparing these disparate 
benefits with a single cost, a wide range of political factors, including the 
mismatch between project lifecycles and budgetary or political cycles, the 
turbulent socio-economic environment around a project that might change 
perceptions of its target and its success, the question of whether it is the pro- 
ject or other parties that are responsible for reaping the benefits, the position 
of the project goals in relation to other related projects, and so on. Often it is 
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not feasible simply to specify a set of well-defined project goals which remain 
constant throughout the project. 

It is in the nature of public projects that there is a wide range of hetero- 
gonous stakeholders, usually with a wide range of heterogonous aims and ob- 
jectives. Stakeholder engagement, establishing success criteria and balancing 
these, for example, is difficult and time-consuming. It can give results that 
the commissioning department might not like, or, indeed, the requirements 
might be mutually conflicting, as in Chapter 4’s fourth vignette concerning 
the freedom of individual cars. Sometimes public debate on the perceived 
need is lacking altogether. In Chapter 6’s Betuweroute case study, “it was not 
made explicit what ‘success’ would mean, not even in tactical terms”. There 
were simply notions of the benefits of extra capacity and strategic advantages; 
indeed, if “success was defined in quantitative economic terms [a specific re- 
port] would undermine the claim that [the project] would be a success”. We 
will discuss this further below. 

For projects in the public view, the immediate is often more pressing than 
the longer term. It is perhaps because of all these difficulties that it so easy to 
concentrate on the well-defined and more easily defensible ‘tactical’ success 
criteria. Estimating and setting tactical targets is known and comfortable, and 
avoids the fundamental question of what we are trying to achieve. Defining a 
project with tactical success measures also gives stability in a world where the 
perceptions and meanings of the objectives might be disputed and changing. 
Moreover, public perception often focuses on the tactical on-time/on-cost 
criteria rather than the strategic project objectives. Certainly it appears to be 
a general phenomenon that, rather than logically concentrating on achieving 
the end result, the public sector often measures success in terms of tactical 
performance — the ‘success paradox’. 

These pressures and issues clearly feed through into the project front-end, 
and muddy the waters even before the start. 


8.4 The early solution 


The ‘paradox of the significance of front-end management’ describes how 
less effort is spent identifying the best conceptual solution than on estimating 
and improving performance against tactical success factors. Public views and 
perceptions are current and pressing now — so it is not surprising that the 
planning horizon is too short, and that short-sighted decisions are made (the 
‘paradox of myopic decisions’). Indeed, often, and perhaps due to political in- 
fluences (such as a minister’s ‘pet project’), we decide on the solution or pro- 
ject upfront, without the logical sequence of identifying the problem that is 
to be resolved, or the needs that are to be satisfied, before exploring solutions. 

Chapter 4 introduces the ‘knowing vs exploring’ trade-off: we need to 
explore the solution space, but how can we explore it unless we understand 
the solution that we are exploring? On the other hand, if we study individ- 
ual solutions too much then we may neglect the exploration. In practice, if 
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we consider a solution, we need a representation that reveals the unknowns 
and complexities, and guides specific knowledge production. The more we 
consider this solution, the more we need to know about it, and so the more 
we focus on it rather than exploring other possibilities. Moreover, this or 
that knowledge will only be applicable to this specific solution and cannot be 
redeployed for evaluating alternative solutions. This implies elaborating par- 
ticular solutions without sufficiently exploring other, different solutions (the 
‘paradox of the opportunity space’), as well as being swamped by too much 
unnecessarily detailed data (the ‘paradox of early information overflow’). 
This trade-off may also provide an indirect explanation for the ‘paradox of 
the significance of front-end management’, since the focus is on developing 
the one solution in the best possible way rather than selecting amongst solu- 
tions about which little is known. 

The discipline of a structured process ought to help ensure that our front- 
end follows a logical process such as that shown in Figure 3.1. As Chapter 3 
discussed, this ought to help lessen the ‘paradox of the opportunity space’, 
since multiple solutions or concepts should be generated by the initial process. 
It ought also to help lessen the ‘paradox of strategic alignment’, as the objec- 
tives and benefits of a project have to be clearly articulated upfront. Indeed, 
there might be a lessening of the ‘predict and provide’ paradox. One im- 
portant recommendation in Chapter 4 was to ensure that the governance of 
the front-end provides conditions that allow responsive actions and changes. 
However, the fundamental trade-off remains, and the experience is clearly 
often that of jumping into a solution too early. 

Sometimes, of course — maybe often — the extreme position is taken of 
adopting one solution at the start without considering any other solutions at 
all. Based on the Norwegian data, Chapter 7 claimed that “most projects start 
out with only one specific conceptual solution to a problem”. This can be 
driven by political considerations. In the C-NOMIS case discussed in Chap- 
ters 2 and 5, the political agenda was considered. In the Betuweroute case 
study in Chapter 6, key actors “had no interest in first discussing the precise 
problems and alternative solutions .... The decision-making procedures did 
not ask for a clear assessment of the problems/challenges, nor for alternative 
solutions”. 

Public projects are clearly highly complex in terms of the meaning of suc- 
cess, the complexity of stakeholders, the definition of the project activities, 
and so on. But public and political decision-making demands simplicity, and 
‘deterministic estimates’ (Chapter 5) as soon as possible. Chapter 5.3 dis- 
cussed two case studies illustrating this dissonance — perhaps going some way 
to explain why early, sometimes very expensive attempts to estimate costs 
and benefits were disregarded (the ‘paradox of disregarded analyses of costs 
and benefits’), and anyway defining individual solutions too early, before the 
solution space was properly explored. 

All of the above points to an early commitment to deciding on a spe- 
cific solution upfront, the results of which have been explored throughout 
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this book. It affects estimation, since costs are considered at an early stage 
based on earlier, lower cost estimates before fully knowing about the solution 
(Chapter 5 talked about the ‘escalating commitment of decision-makers’). 
Chapter 6 described this ‘lock-in’ effect, which might be caused by political 
decisions, or reliance on early (too-low) cost estimates, followed by escalat- 
ing commitment, the need for justification, inflexibility and the closure of 
alternatives. 

Even here, though, the solution is not simply to avoid early solution choice 
in all circumstances, because the project owner is not a passive figure seeking 
a solution, but rather an active participant. Chapter 4’s ‘evaluation vs shap- 
ing’ trade-off discussed the compromise between evaluation and leveraging 
a solution to shape the future. Major projects are always subject to significant 
uncertainty and lack of knowledge, so evaluation of options can only go so 
far. Project sponsors often therefore focus on shaping the world around the 
project, and project representations become a future-making tool. This per- 
haps explains, and to some extent justifies ‘the paradox of the opportunity 
space’ and also the ‘paradox of the significance of front-end management’: 
endless evaluation does not always promote a better project, but sometimes 
the rapid selection of a solution — if not the best solution — may, in fact, be 
desirable because it facilitates the structuring effect on the environment. 

Chapter 7 recommended an ex ante review at the end of the front-end 
before a project actually goes ahead, and offered some guidance. After these 
chapters were written, the UK National Audit Office (2021) published a syn- 
thesis of how they review major projects, which could provide useful guid- 
ance, particularly for ex ante reviews. A strong governance process should 
create a ‘stop-go’ hurdle before the project proceeds, such as the government 
decision in Chapter 3, particularly if it includes the requirement for such an 
ex ante evaluation, as in the Norwegian system described in Chapter 7.4. 


8.5 Stakeholders and consultation 


Chapter 2 described the wide range of stakeholders that may be involved in 
a project. Some will be powerful elements within government; some will be 
disinterested regulators or permission granters; there may be some who feel 
powerless, but have important views to express in what is ultimately a public 
project; pressure groups might consider themselves to be involved; hence 
there is likely to be a hierarchical range of analysts and decision-makers. Un- 
like much of private industry, the process is (or at least should be) carried out 
in the public view — and with the knowledge that there could be public and 
parliamentary scrutiny after the event. The public, particularly, can be very 
vociferous in expressing their views about such projects; the use of the word 
‘uproar’ in Chapter 4’s third vignette, when a particular element was left out 
of a public presentation is not uncommon. 

The discussion above and in Chapter 2 showed multiple issues in gaining 
input from a range of heterogenous stakeholders with possibly conflicting 
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views. This need to consult stakeholders is discussed in Chapter 4’s ‘pluralism 
vs support’ trade-off. There are clear problems in trying to gain a complete 
pluralistic scrutiny of the front-end from all stakeholders — this muddies the 
water to such an extent that a solution may not be found, or may prevent 
supporters from rallying around the successful option, and offer arguments 
for opponents. Again, this is a difficult balance in a public-facing project: the 
need to consult fully against ‘paralysis by consultation’. One suggestion in 
Chapter 4 followed the observation that innovative technologies, including 
systems such as Building Information Modelling (BIM) are already used to 
develop product designs, and “an unexploited advantage of these technolo- 
gies may be the possibility to enable collaboration with stakeholders at the 
very early stage of a project”. 

Rarely will any of these conflicting views be from disinterested view- 
points, as stakeholders want specific benefits, but few will be paying for these, 
or paying for over-spends, which can lead to the ‘paradox of perverse incen- 
tives’. This is where projects which lay no financial obligations on the target 
group may cause perverse incentives — we will revisit this below. 


8.6 Information 


A key aspect of the front-end being a process rather than a point in time — a 
very long process in some cases, such as many military projects — is that we 
have to consider the temporal flows of information. 

Running through all the chapters (see Figure 8.1) is the ‘paradox of early 
information overflow’. Rather than carefully assessing, and making judge- 
ments on selected key information, there is a deluge of information upfront, 
all being very early indications, and decisions are based on these. This leads to 
early solutions, as highlighted above. As stated in Chapter 7, “in many cases, 
the amount of specific, detailed information contributes to restricting the 
original choice of concept to the extent that it will eventually be the realised 
option”. This illustrates what happens in practice with Chapter 4’s ‘knowing 
vs exploring’ trade-off — it is not just that by exploring individual solutions 
too much, we may neglect the exploration — in fact, this is what happens in 
practice. There is merit in making strategic decisions on ‘scant information’ 
(see Williams et al. 2009). Chapter 7 discussed how early decision-making 
should consider the problem in its context of stakeholder interests — “rather 
than being a hindrance, lack of detailed information early on can actually be 
a benefit, providing focus and flexibility to the analysis”. 

As the front-end proceeds, Chapter 5 described how new and better infor- 
mation becomes available. In addition, Chapter 7, in particular, considered 
the validity of information over time. While some data might remain robust 
over the period of the front-end, demand data, for example, might change 
over a short period. This is particularly true of projects with a longer front- 
end, such as military projects. In the well-known torpedo battery example, 


Conclusions 199 


which was “officially opened as planned and without cost overrun, it was 
closed down one week later by Parliamentary decision” (Williams & Samset 
2010, p. 40). An ex ante review before the project was executed might surely 
have understood the out-of-date premise for the project. 

Indeed, the public conception of rational decisions made at a single point 
of time in the front-end is clearly at odds with the actuality of humans’ 
sense-making, both as they explore options, but particularly as they seek to 
know more about particular options. Chapter 4 talked about the “improvi- 
sational, bricolage-like nature of representation construction’, discussing not 
only how the information comes to hand, but also how the mechanisms by 
which it is represented can have considerable impacts upon the process of 
understanding. 

This dispassionate discussion of information presumes that the main char- 
acters are disinterested and unswayed by external motivations. Chapter 4 
already noted the need to — and benefit of — “consider[ing] a broader range of 
cognitive, emotional and social reactions”. However, the work of Flyvbjerg 
(2003 and following) would point to ‘strategic mispresentation’, as political 
or other motivations seek to influence the information and particularly the 
estimates put forward. Chapter 5 discussed how, 


this may manifest itself in assumptions that may be best described as 
‘underdeveloped’ and optimistic forecasts of future long-run bene- 
fits. Whilst an unrealistically low initial cost estimate may increase the 
chance of the project being funded — future problems are “baked in’ and 
are often irreconcilable. 


Examples of this in the public sector are legion. Within this book, the ex- 
tended Betuweroute case study in Chapter 6 described some of the politi- 
cal estimation of cost, with one official not sleeping for fear that politicians 
would read some specific, disinterested, better — and much higher — cost 
estimates and “a positive decision to build would be endangered”. The final 
cost was, indeed, close to those disinterested estimates. One issue, as Chapter 
6 pointed out, is that “those who benefit from a positive decision to build are 
not those who need to pay”, which leads to the question of accountability. 


8.7 Accountability 


Chapter 2 discussed the multitude of stakeholders and their differing needs 
and desires. However, this only considered what they wanted out of a project, 
rather than their input, Chapter 6 bringing to the fore the ‘paradox of per- 
verse incentives’ for those actors who do not have to contribute. 

In a public project, there are many actors with many motivations, who 
mostly do not have to pay if the project goes wrong. For example, “the Port 
of Rotterdam supported the project but did not have to pay, so it was easy for 
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them to ask for the line” (Chapter 6). Furthermore, those responsible for pro- 
viding estimates can make these unrealistically low (or benefits predictions 
unreasonably high) since they do not pay for the results: 


the paradox of perverse incentives explains that it made sense for pro- 
ponents of the Betuweline to come with excessively low cost estimates. 
Next a process of lock-in occurred, as a result of which there was no way 
back, long before the decision to build. 


Indeed, Chapter 6 showed how this underlying paradox can loop back, help- 
ing to explain several of the other paradoxes (which would start to make 
Figure 8.1 more realistic but somewhat difficult to read). Perverse incentives 
can be so strong that an actor can be “motivated to make choices resulting 
in a project that is a complete failure seen in retrospect” (Samset & Volden 
2016, p. 308). 

Therefore, a key governance question in such projects is: who is account- 
able for the outcome of the project? In the language of Chapter 2, responsi- 
bility for delivery of the as-defined project output (i.e. the ‘tactical success’) is 
in the hands of a project manager, and often there is a contract with a private 
sector partner. However, as Chapter 2 explained, the public is interested in 
the overall strategic success, in other words, delivery of some useful contri- 
bution to the life of the country, at a reasonable price. There is therefore an 
increasing realisation that there has to be accountability within the system 
for the project outcome — the benefit that the project brings to the country. 

In the UK system there is a position known as the “Senior Responsible 
Owner’ (SRO) who is responsible to the government and to parliament for 
delivery of both the project and the benefits, and who has to sign a letter 
agreeing to this (see UK Government 2019 for an example). These responsi- 
bilities are set out in a UK Government handbook (Infrastructure and Pro- 
jects Authority 2019), which clearly states, “The senior responsible owner is 
accountable for a programme or project meeting its objectives, delivering the 
required outcomes and realising the required benefits. The senior responsible 
owner of a government major project is accountable to parliament”. 

This is easy to state, but there are clear difficulties which we have already 
established. 

As pointed out in Williams et al. 2020, the evaluation of benefits can be 
difficult to disentangle from the general movements of the economic envi- 
ronment; there is often a move away from the original pre-defined project 
(launched under a previous national budget and maybe even a different na- 
tional government) and “the emergent and sometimes fluid nature of bene- 
fits”. Furthermore, some projects only facilitate the gaining of benefits, while 
other bodies are required to ‘harvest’ those benefits. For example, a piece 
of infrastructure might facilitate economic development, but only if the re- 
gional authorities take advantage of the project to make that development; 
supply of an IT system might make working with government easier, but 
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only if it is utilised. Sometimes the causal route between the project output 
and the desired benefit is long and difficult to justify, for instance between 
the building of new prisons and the reduction of re-offending (see Chapters 5 
and 2). 

But it is not the officials themselves who actually execute the project. So 
far we have discussed only the public sector and the demands upon decision- 
makers. The project itself will be executed by the private sector, so at some 
point there needs to be a contract signed with a company or consortium — 
again trying to avoid ‘perverse incentives’. This might be easy in the hypo- 
thetical situation where tactical success factors are easy to define and are com- 
pletely aligned with the strategic aim of the project, which remains constant 
throughout. However, we have already seen that none of this is likely to be 
the case, and that some sort of partnership needs to be formed with the pri- 
vate sector. There are also likely to be other private companies with a strong 
interest in the project, even if they are not part of the project execution team. 

Identifying and allocating risk within such outsourced contracts is often 
complex, and our understanding of these is having to develop to ensure that 
the public sector has appropriate contracts (Bloomfield et al. 2019). Even if 
we can identify the risks unambiguously, there is a trade-off between allo- 
cating responsibilities and risks between participants and enhancing collab- 
oration. In Chapter 4, seeing project representations as ‘boundary objects’, 
the more these are defined prescriptively, the easier it is to allocate respon- 
sibilities, but the harder it is to produce collaboration. The solution to this 
recently has been collaborative governance and contractual forms — but the 
requirement for these often logically implies the selection of, or at least con- 
vergence into, a conceptual solution early in the front-end process (‘paradox 
of the significance of front-end management’). Again, Chapter 4 looked at 
relational contractual arrangements which allow responsive actions to obsta- 
cles or ‘real life’. 


8.8 Reflecting on the project 


As we have said above, public projects are carried out with a certain degree of 
transparency, and the public arena will want to know whether they have been 
given value for taxpayers’ money. The ‘public arena’ in this case includes not 
just the general public — who may have particular slants on their views (see 
Chapter 2). It also includes formal auditing organisations (in the UK, the Na- 
tional Audit office) as well as governmental or parliamentary bodies (in the 
UK, this includes parliamentary committees). Indeed, Chapter 5 referred ex- 
tensively to one series of hearings by a UK Parliamentary Select Committee. 

An evaluation should stand back and consider the project against the suc- 
cess criteria, as laid out generically in Chapter 2, and again in Chapter 7.3. 
This was done in Norway by the Concept programme (as described in Samset 
& Volden 2016), with results laid out in Chapter 7. Achievement of cost/time 
targets can be evaluated quantitatively, although this can be problematic if 
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the final output differs from (or in some cases bears little resemblance to) the 
original plan, perhaps because circumstances changed during the project — 
which is not unusual for public projects. However, higher level criteria such 
as ‘effectiveness’, ‘relevance’ and ‘sustainability’, which might be difficult to 
quantify (let alone monetise), have to be largely evaluated subjectively. 

There are significant problems with larger public projects in evaluating 
benefits ex post (see, for example, Williams et al. 2020). For projects hav- 
ing an economic impact, the effect is often indistinguishable from general 
changes in the economy (i.e. it is difficult to evaluate what would have hap- 
pened without the project). As described above, some projects only facilitate 
benefits, leaving other bodies to ‘harvest’ those benefits. Where projects are 
part of a portfolio, it can be difficult to disentangle the effect of individual 
projects. This means that the clear, unambiguous allocation of benefits to a 
particular project might be very difficult in some circumstances and requires 
comparison with hypothetical counterfactual options. 

However, despite all this, Chapter 7 shows a process which is operating 
well and has the capacity to contribute to greater delivery of projects, as well 
as ameliorating the effects of all ten paradoxes. 


8.9 In conclusion 


In conclusion, in this book we have looked at the reality of the genesis plan- 
ning, launching and delivery of major public projects. There is plenty of ad- 
vice and guidance for the public decision-maker, but actual practice appears 
not to be so simple. As Samset and Volden (2016) showed a few years ago, 
there seemed to be a number of curious ‘paradoxes’ causing projects to be 
launched in ways that were later seen as not of the best. Figure 8.1 shows these 
paradoxes and the way in which many flow from each other. 

Looking more deeply into the front-end of the project showed a number of 
fundamental trade-offs in Chapter 4 which to some extent are unavoidable. 
It is important that a project understands, acknowledges and manages these 
trade-offs and steers a clear course. 

Consideration of the ‘paradoxes’ has enabled us to understand them better, 
as well as the underlying causes — both from the environment and from the 
actors. Some aspects are incorrect behaviours that need to be understood and 
avoided. Some, however, need to be understood and managed as paradoxes, as 
argued in emerging paradox theories such as Schad et al. 2016 (see Chapter 4). 

This chapter cannot do justice to the depth of each individual chapter, but 
has noted some themes which cut across all chapters. 

The authors of the book hope that their discussions will help to produce 
more clarity for decision-makers — as well as public understanding of the 
decisions being made — so that some behavioural traps can be avoided, better 
decisions made in paradoxical situations, and so that we can plan and deliver 
projects that actually provide our countries with the benefits they need effi- 
ciently and effectively. 
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Figure 8.1 Chapters and paradoxes. 
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1 Introduction 


A large and increasing share of the activities taking place in private as well 
as the public sector is organised as projects. In private sector projects, the 
ultimate goal is to improve the company’s profitability, either directly or 
indirectly, through improvements in its competitiveness. In public projects, 
the commissioner is the government, representing the entire society and its 
taxpayers. In such cases, the benefits of the project must be considered in 
a broader societal perspective, to ensure that the project provides value for 
money and contributes to the desired development. 

There are many challenges facing public investment projects that must be 
overcome to achieve project success, such as lack of competence among plan- 
ners, avoidance of hidden agendas during planning, underestimation of costs 
and overestimation of benefits, unrealistic and inconsistent assumptions, and 
how to secure essential planning data and adequate contract regimes. Many 
of these problems can be interpreted in terms of deficiencies in the analytical 
or political processes preceding the final decision to go ahead. Hence, the 
importance of the front-end decision-making phase must be recognized to 
strengthen project governance. 

The term governance is derived from the Latin word gubernare, mean- 
ing ‘to steer’. It refers to the administrative and process-oriented elements 
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of governing, whether undertaken by a government, market, or network, 
whether over a family, tribe, formal or informal organization, or territory, 
and whether through laws, norms, power, or language (Bevir, 2013). Gov- 
ernance is about processes of rule more than institutions of government. It 
relates to processes and decisions that seek to define actions, grant power, and 
verify performance. Different instruments are available to improve govern- 
ance, ranging from legally binding regulations, to economic and other types 
of incentives, as well as information and skill development. The challenge in 
governance is to identify the optimal mix of different instruments. 

Project governance refers to the processes, systems, and regulations that the 
financing party must have in place to ensure that projects are successful. This 
would typically include a regulatory framework to ensure adequate quality 
at entry, compliance with agreed objectives, management and resolution of 
issues that may arise during the project, and standards for quality review of 
key appraisal documents (Samset and Volden, forthcoming). These processes 
and regulations can often be described in terms of stage-gate phase models. 

Project management refers to the processes established to organize and 
manage resources required to complete a project within defined scope, qual- 
ity, time, and cost constraints.W hereas the literature on project management 
is substantial, project governance has only recently become an issue of impor- 
tance in the project management community (e.g. Müller, 2009). 

Peter Morris (1994) brought to our attention that in the early years, project 
management had an extremely narrow focus, reflected only in the project 
life cycle, and ignoring the critical front-end. He noted that as long as we 
only focus on the life cycle itself, we are missing the critical front-end and 
institutional elements (shown in his Management of Projects paradigm) that 
more accurately typify the responsibilities of the project owner and the pro- 
ject manager. 


2 The present study 


In the year 2000, the Norwegian Ministry of Finance introduced a govern- 
ance regime for the country’s largest public investment projects, the so-called 
Quality Assurance (QA) regime, in terms of a mandatory quality-at-entry 
scheme to meet such challenges. It is a simple stage-gate process with a top- 
down review of the quality of project proposals, which are typically the result 
of bottom-up processes of analysis and decision making in society. The Nor- 
wegian QA scheme includes two external reviews in the front-end: Quality 
Assurance of the conceptual solution (QA1) before Cabinet decision whether 
to start a pre-project, and Quality Assurance of the cost and steering frames 
(QA2) before the project is submitted to Parliament for approval and funding 
(see Fig. 1). 

In parallel to the QA regime, the Norwegian University of Science and 
Technology in year 2002 initiated the so-called Concept research pro- 
gramme, designed to focus on the front-end management of major public 
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Cabinet Parliament 
decision approval 
Needs  Pre-study Pre-project Project ) Effects 
Quality assurance of Quality assurance of 
conceptual solution cost and steering frames 


Figure 1 The Norwegian Quality Assurance regime, a stage-gate phase model with 
two external reviews in the front-end of major public projects. 


Source: Norwegian Ministry of Finance. 


projects. The governance scheme clearly would be a unique laboratory for 
research on longitudinal data. It has allowed researchers to follow the larg- 
est public projects in Norway since 2002. The Concept programme works 
to develop the research frontier in the area of project governance. This is 
undoubtedly an interdisciplinary field, and the programme has conducted 
separate studies in areas such as public management, project management, 
portfolio management, economic analysis, planning, decisionmaking, risk 
analysis, contract management, the theory of incentives, applied logic, and 
judgmental assessment. 

The idea was to broaden the perspective on projects. To quote Morris 
(2009:60), “effective management of projects is more than just execution- 
oriented project management. Projects are undertaken to create value and 
deliver benefits. Shaping the interaction between the sponsor’s goals and the 
way the project (or programme) is to be developed, in the best way possible, 
absolutely crucial — probably one of the most important aspects of managing 
a project”. 

This understanding is an underlyingmotivator of our research. However, 
the approach has been inductive rather than deductive. It has been more of a 
probe into new areas than a process guided by precisely formulated and the- 
oretically founded problems. The perspective has been on projects as means 
to create value and deliver benefits. Some studies had a focus on decisions, 
others on analysis, but all of them were meant to provide insight into what is 
here termed project governance. 

Miller and Lessard (2000) contended that the front-end phase from incep- 
tion and until the budget is approved by Parliament takes 6-7 years on aver- 
age in major public investment projects. This is also the case in Norway. The 
subsequent implementation phase takes typically 3-5 years, and projects will 
have to be at least three years into their operational phase before an ex post 
evaluation can be undertaken. The Norwegian QA scheme has now been in 
operation for 14 years involving about 20 new projects each year. This means 
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that there is a tremendous time lag as regards availability of empirical data. 
During the early years of the programme, researchers were first left with the 
option to do theoretical desk studies, and then gradually shifting into more 
hands-on studies of procedures and practices in planning, quality assurance 
and decision making during the front-end phase as more projects were added. 
Only recently, empirical data are becoming available. At present, the total 
number of projects is about 260, of which only 50 have been implemented so 
far. In addition, not more than 10 projects have reached a degree of maturity 
that allow for ex post evaluations. 

Fourteen years after the quality assurance scheme was introduced it has 
proved to have a positive impact on cost control, since almost 80% of the first 
40 projects were completed below budget, which 1s quite remarkable (Samset 
and Volden, 2013a, 2013b). Getting to grips with the choice of conceptional 
solution and securing the strategic performance of projects may prove to be a 
much more complex matter. It is challenging, but no less interesting. 

The present study draws on some research findings from the Concept pro- 
gramme, presented below in terms of ten paradoxes, all of which have impli- 
cations for the theory of project management and project governance, as well 
as a concluding study that demonstrates some of their implications. The term 
“paradox” in this paper is used to describe situations with a counter-intuitive 
result, some of which are based on fallacious reasoning or incomplete or 
faulty analysis. These are the paradoxes of: 


How success is understood 

The significance of front end management 
Early information overflow 

The opportunity space 

Strategic alignment 

Cost estimation 

Disregarded analyses of costs and benefits 
“Predict and provide” 

Perverse incentives 

Myoptic decisions. 
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Their common denominator is that they all focus on the choice of conceptual 
solution. Each paradox is rooted in one or more studies in the programme, 
but also inspired by research findings presented in the biannual international 
Concept symposia on project governance. According to Pinto (2014), these 
symposia “... have provided much of recent theoretical and epistemological 
structure to the construct of project governance. Papers from these symposia 
have been on the leading edge of many of the insights we have on the current 
state of project governance”. 

The three first paradoxes are not rooted in empirical research, but in 
desk studies and literature reviews. The remaining seven (numbers 4—10) 
are based on cased studies involving 5—40 cases, most of them are major 
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public projects that have been subjected to external quality assurance under 
the Norwegian QA scheme. It should be noted that most of these studies 
are written in Norwegian only. The reports can be downloaded from the 
programme’s website www.concept.ntnu. no, where also summary reports 
in English are available. 

Since this paper can only provide brief snapshots of the studies, most of the 
references will have to be found in the underlying reports, and are only to a 
limited degree included in this paper. 


3 The success paradox: success is measured in terms of 
tactical performance rather than strategic performance 


The term “success”, used as an indicator, is a highly complex and aggregated 
measure. More than two decades ago Pinto and Slevin (1988) concluded that: 
“the concept of project success has remained ambiguously defined both in the 
project management literature and, indeed, often within the psyche of pro- 
ject managers... Until project management can arrive at a generally agreed 
upon determinant of success, our attempts to accurately monitor and antici- 
pate project outcomes will be severely restricted”. 

“Success” may be interpreted differently by various individuals and in- 
stitutions. It may be measured differently in different types of projects, and 
different individuals tend to assess the success of the same project differently 
depending on their preferences, values and to what degree they are affected 
by the project. In addition, the degree of success is time-dependent. For in- 
stance, Shenhar et al. (2001) offer a chronological sequence of events as a 
compound definition of project success: (1) meeting time, budget, and other 
requirements, (2) impact on the customer, (3) benefit to the performing or- 
ganization, and (4) preparing the future. The project’s stakeholders do not 
necessarily share the same view of success. The project manager typically sees 
his job successfully accomplished when the project is done on time, within 
budget, and to specifications. The users will be concerned about the imme- 
diate effects of the project, and the investor or commissioner will typically be 
more concerned with the long-term economic viability. 

Success as a generic term means to gain advantage, superiority, accom- 
plishment, achievement or added value. Measuring success will have to look 
beyond the immediate outputs of the project to assert the anticipated and 
wider impact in a longer-term perspective. A hospital will ultimately have 
to be assessed in terms of its health benefits. An industrial project might be 
judged essentially in financial terms, and an infrastructure project in term of 
its utility. 

The assessment of success can be in absolute or in relative terms — that is 
in relation to what was agreed versus what was realistically achievable. Am- 
bition is expressed in terms of the project’s stipulated objectives. Its effective- 
ness is a direct measure of what has been actually achieved. Clearly, success 
measured in absolute terms may give a misleading conclusion if objectives are 
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unrealistically ambitious. By measuring in relative terms, that is in relation to 
what could reasonably be expected as compared with experiences in similar 
projects — the same project might possibly be considered a success. 

The media tend to give unsuccessful projects more publicity than success- 
ful ones. However, their perspective is highly restricted. The number one 
criterion of failure in the media is cost overrun; number two is delay in time. 
Truly, a much wider view needs to be taken on the success and failure of 
projects. The initial choice of project concept is of critical importance. This 
represents the one key decision of many made during the lifetime ofa project, 
which is likely to have the largest impact on long-term success or failure Wil- 
liams (2008). Here, by “the project concept” we mean much more than just 
the technical solution — it includes the entire business case, all of the various 
organisations involved, and the various mechanisms and arrangements in- 
volved in the inter-organisational relationships, see Miller and Hobbs (2009). 

Here, it is necessary to distinguish between the projects’ tactical and stra- 
tegic performance. Success in tactical terms typically means meeting short- 
term performance targets, such as producing agreed outputs within budget 
and on time. These are essentially project management issues. Strategic per- 
formance, however, includes the broader and longer-term considerations of 
whether the project would have a sustainable impact and remain relevant and 
effective in its operational phase, throughout its lifespan. This is essentially a 
question of getting the business case right, or, in short, of choosing the most 
viable project concept. 

This is illustrated in Fig. 2. Tactical performance is a question of how the 
project is implemented, i.e. how inputs are converted into outputs. These 
are measures of its efficiency, here measured in terms of the cost, timing and 


Project governance 


Strategic Rele- Sustain- 
performance vance ability 


Tactical 
performance 


Project management 


Figure 2 Successful projects. Tactical performance is a question of delivering the pro- 
ject outputs as planned, while strategic performance is the worth or utility 
of the project as seen in a long-term perspective (Samset, 2014). 
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quality of deliverables. Strategic performance is a question of how the project 
performs after the outputs have been delivered. This will have to be mon- 
itored with the more compound measures mentioned above, which would 
cover the broader and longerterm perspectives and to a lesser degree involve 
focusing on technology and management issues, but more on societal and 
economic aspects. 

One example of tactically inefficient projects but viable in strategic terms 
could be the University Hospital in Oslo, Norway. Due to emerging new 
technologies and added responsibilities, captured during the engineering 
phase after the budget was decided, it was completed a year behind schedule 
and with considerable cost overrun, adverse newspaper reports and a public 
inquiry. No doubt that cost overrun was considerable in absolute terms, but 
in relative terms, it was equivalent to only a few months’ operational costs for 
the hospital, and therefore insignificant in a lifetime perspective. The overall 
conclusion after a few years of operation was that the University Hospital 
was a highly successful project; and it would perhaps be unfair to suggest that 
initial decisions should be able to capture problems at this level of precision. 

More serious by far is when a project fails in strategic terms, even if it suc- 
cessfully produces the intended outputs. It means that the choice of concept 
turns out to be the wrong one in relation to the problem at hand. In some 
cases, it may create more new problems than it solves, in others the initial 
problem no longer exists once the project is completed. One such example is 
an on-shore torpedo battery built inside the rocks on the northern coast of 
Norway in 2004 (Samset, 2008a, 2008b). The facility was huge and complex, 
designed to accommodate as many as 150 military personnel for up to three 
months at a time. It was officially opened as planned and without cost over- 
run. Already one week later it was closed down by Parliamentary decision, 
since it was obvious to all involved that a potential enemy would not expose 
its ships to such an obvious risk; the concept had long since been overtaken by 
political, technological and military development. What was quite remarka- 
ble was that this project, which can only be characterized as a strategic failure, 
got much less negative attention in the media than the University Hospital, 
possibly because it was a success in tactical terms. 

Clearly, a successful project is one that delivers its outputs and significantly 
contributes to the fulfillment of agreed objectives. Moreover, it should have 
only minor negative effects, its objectives should be consistent with needs 
and priorities in society, and it should be viable in the sense that the intended 
long-term benefits resulting from the project are produced. These require- 
ments were first formulated for USfunded international development projects 
by the United States Agency for International Development (USAID) in the 
1960s, and subsequently endorsed by the United Nations (UN), the Organ- 
ization for Economic Co-operation and Development (OECD), and the Eu- 
ropean Commission (EC) (USAID, 1980). They comprise five requirements 
or success factors that have to be fulfilled, i.e.: the project’s efficiency, effec- 
tiveness, relevance, impact and sustainability. These are tough requirements 
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that go far beyond the issues that usually are covered by the media or indeed 
by many planners and decision-makers. 

Applied as standard requirements both up-front and ex post when projects 
are evaluated would be likely to improve project governance considerably in 
the future. 


4 The paradox of the significance of front end 
management: less resources are used up front 
to identify the best conceptual solution (project 
governance), than to improve tactical performance 
during implementation (project management) 


Projects are exposed to uncertainty in varying degrees and this is often used 
to explain their failures. Uncertainty characterises situations where the actual 
outcome of a particular event or activity is likely to deviate from the estimate 
or forecast value. Uncertainty may have many and various causes, related to 
the situation itself: the design of the project, the time perspective, available 
information, the implementation of the project, etc. (Marshall and Ritchie, 
1993). Obviously, decisionmaking becomes difficult when uncertainty is 
high. Availability of relevant information reduces uncertainty from the deci- 
sionmaker’s point of view. It is widely believed that uncertainty is highest at 
the initial stage, when the project concept is conceived, and that it tends to 
reduce rapidly as information accumulates over time. 

This line of thought is illustrated in Fig. 3. It follows that the utility of 
adding information is at its highest in the earliest stage. It is also commonly 
believed that the decision-maker’s flexibility and the cost of making amend- 
ments are opposites. This is visualized with a similar graph. Decision-makers 
can juggle with different ideas and strategic solutions to a problem in the 
initial stages, but once decisions are being made, essential choices become 
locked, and it is more difficult and expensive to change the overall design. 
Therefore, major issues such as agreeing on the most effective solution to a 
problem and the choice of concept need to be dealt with as early as possible — 
later on is too late. Less essential issues such as avoiding major cost overrun 
can be handled later, for example when the final budget is agreed. 

In Fig. 4, the distinction is made between the front-end and the implemen- 
tation phase. The graph suggests that the potential to reduce uncertainty and 
risk is the largest up-front, and decreases substantially when the project is im- 
plemented. It is a paradox therefore that most of a project’s planning resources 
may be spent on detailed planning and engineering, while too little is usually 
spent on getting the idea right from the start where the potential to reduce 
uncertainty by means of adding information is the largest. The paradox is that 
most resources are used to reduce uncertainty during the implementation 
phase, where the potential is much less. 

Recent literature has highlighted the front-end phase including the project 
definition, as important for ensuring strategic project success (see for example 
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Figure 3 People’s conception of how uncertainty is affected by information and how 
flexibility to make amendments is restricted by cost, as time passes in a 
project. 

Source: Authors. 


Merrow, 2011; Morris, 2013). Where projects fail strategically, it is likely that 
the problem can be traced back to decisions in the earliest phases, when the 
initial idea was conceived and developed. What happens during the front- 
end phase is therefore essential for a project’s success. A study by the World 
Bank based on a review of some 1125 projects concluded that 80% of the 
projects with a satisfactory “quality-at-entry”! were successful, while only 
35% of those with an unsatisfactory quality-at-entry achieved success (World 
Bank, 1996). Improved front-end management is therefore likely to pay off 
in a wider life cycle perspective, as evinced by the IMEC study (Miller and 


1 Quality-at-entry was used as an indicator to characterize the identification, preparation 
and appraisal process that the projects had been subjected to upfront. 
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Figure 4 The project life cycle. Uncertainty is greatest in the front-end phase and 
diminishes as more and better information is acquired for making decisions. 
Source: Samset (2010). 


Lessard, 2000). One way of improving quality-at-entry is by challenging ini- 
tial ideas and applying simple analyses, extracting and making use of previous 
experience from similar undertakings, and consulting with stakeholders. Jor- 
dan et al. (1988) argued that 15% of the time and resources in projects should 
be spent on front-end work, whereas Miller and Lessard (2000) suggested up 
to 35%. 

In most cases the key issue at the earliest stage is to shed sufficient light on 
the underlying problem that provides the justification for the project, and the 
needs that the project is meant to satisfy. Detailed information about possible 
alternative solutions is less relevant. This illustrates what seems to be a major 
dilemma, since most projects originate as one specific solution to a problem, 
while the problem itself may not be analysed sufficiently, and alternative solu- 
tions may not have been considered at all. Typically, the preferred concept 
originates in the mind of one individual, based on intuition and experience, 
rather than systematic analysis of problems, needs, requirements, etc. Most 
of the information generated is associated only with the initially identified 
solution (Whist and Christensen, 2011). 

A second dilemma is that this information, which may be very detailed and 
specific, tends to lock decisions into the initially preferred concept — to the 
extent that this will inevitably be the one that is finally chosen. It is all too 
rare that alternative concepts are identified and analysed to the extent that 
they get a fair trial in the subsequent decision process. 
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5 The paradox of early information overflow: decisions 
are based on masses of detailed information up front 
rather than carefully selected facts and judgmental 
information relevant to highlight the essential issues 


It follows from the above that the front end phase is when fundamental 
choices are made, uncertainty is at its highest freedom to choose is at its 
optimum, and available information is most restricted. Adding information, 
therefore, makes sense — but only to a certain degree. The crucial issue is not 
the volume but what type of information is needed. 

But contrary to the idea depicted in Fig. 3 the sheer amount of available in- 
formation upfront might not be the issue. In the initial phase of a project the 
priority is to establish an overall perspective, and to analyse the problem in 
its context, considering the needs and priorities of stakeholders, users and af- 
fected parties, in order to come up with a sensible strategy. Opportunities and 
risks should be considered. Experience suggests that creativity, imagination 
and intuition can be more valuable at this stage than large amounts of data. 

Decision making may be complex, unstructured, and affected by chance. 
Analysis may be biased or inadequate. Decisions may be affected more by 
political priorities than by rational analysis. Political priorities may change 
over time. Alliances and pressures from individuals or groups of stakeholders 
may change. The amount of information is large and may be interpreted 
and used differently by different parties. The possibility for disinformation is 
considerable. 

Another aspect is that the early selection of a concept tends to survive 
decision-making, regardless of process, expertdriven rationalistic or more 
open-ended and democratic. This makes a strong case for proper research to 
identify the most viable concept up front. However, time factor, complexity 
and lack of predictability also imply that the outcome of rationalistic plan- 
ning upfront tends to alter over time. 

Exact quantitative information tends to be more affected by time than the 
choice of concept. On the one hand it is obvious that the higher the preci- 
sion, the more rapidly information is outdated.? It is tempting to speak of the 
“half-life of information”, see Fig. 5. For instance, exact information about 
the demand in a fast-developing market will have limited value after months, 
or even weeks. On the other hand, there are many examples to suggest that 
qualitative assessments tend to remain valid for much longer. Consider the 
assessment of users’ fundamental preferences within a market segment. While 


2 We need of course to make a distinction between lasting information, for example physi- 
cal data on the one hand, and less durable information such as economic estimates on the 


other hand. 
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Figure 5 Half-life of information. Validity tends to decrease over time during the 
front-end phase. More rapidly for accurate data than for less accurate 
estimates. 

Source: Samset (2010). 


it might not be possible to make a valid prediction of the actual demand three 
years into the future, it may be judged that demand will continue for a long 
time and can therefore be relied upon in strategic planning up front. 

This suggests that restricted quality of information upfront may not be a 
major problem, since the need for precise information is low. It increases as 
the time for detailed planning approaches. In other words, the utility of exact 
information tends to reduce with the time-span. The opposite seems to be 
more of a problem: when decision-makers are confronted with an abun- 
dance of detailed information at an early point in time it may result in what 
is referred to as “analysis paralysis”. This problem is discussed by Williams 
(2008). And besides, the cost of collecting information on a specific topic 
usually increases progressively with the amount of information collected. 
This is because more information requires more in-depth studies or more 
wide-ranging information searches. On the other hand, the gain in utility 
of additional information tends to decrease. This is because there is usually a 
critical amount of information that is needed to get the necessary insight in 
a situation: Additional information will be of limited use. Maximizing the 
utility/cost-ratio will set a limit to the amount of information that is useful 
(Jessen, 2012). 

This emphasizes the need to invest in relevant information at the earliest 
stage of a project, while at the same time limits the search to what is useful 
for decision-making at this stage. A targeted search for information regarding 
the main uncertainties likely to affect the project is more cost-effective than 
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an unguided search, since it makes it possible to increase the share of relevant 
information and reduce the total amount. 


6 The paradox of the opportunity space: the choice of 
conceptual solution is made without systematically 
scrutinizing the opportunity space up front 


Every project is initiated to solve some problem or meet some needs. And 
every project faces a choice of concept in terms of how to solve this problem. 
Consequently, a key task in the early phase of a project is to identify possi- 
ble ways to solve the problem it has been mandated to solve (setting up the 
opportunity space), furthermore to evaluate alternative concepts (limiting 
the opportunity space), and decide on the one best suited. There is much 
evidence to suggest that this is not always how things are done. 

One problem is that planners are discipline experts with an inherent ten- 
dency to emphasize some aspects of the matter and downplaying others. The 
same may apply to organization undertaking the planning; its rules, pro- 
cedures, etc. This is the reason for path dependency (Dosi, 1997; Margolis 
and Liebowitz, 2000); systematically choosing some solutions while avoiding 
others, even if these conflict with rational choices. 

The situation become even more complex since these decisions are made 
at the intersection between the professional and political, in other words in- 
between what is rationally sound and politically possible. In the end, the 
complexity of the decision situation depends very much on whether there is 
an agreement about what one wants to achieve and what are the best means 
to this end (Christensen, 1985). 

A case study of 17 major public projects was carried out to explore the 
use of the opportunity space, i.e. how it was defined, the type of conceptual 
alternatives identified and the effect on decisions (Andersen et al., 2014). It 
was found that in 11 cases the choice of concept had in reality already been 
made when the front-end process started, only in six cases, truly unique al- 
ternatives were identified. In most projects the analytic focus was narrowed 
to detailed project-specific issues at the expense of overall societal aspects. In 
half of the projects, the opportunity space was restricted to such a degree that 
real alternatives were excluded. There was a strong degree of path depend- 
ency where the alternatives represented a continuation of the current solution 
or variations over a theme. 

It was emphasized that these processes take place on the borderline be- 
tween the professional and political spheres, especially since the political 
backdrop is what exerts the most restricting effect on the opportunity space. 

While the analytical process is largely within the realm of the professional 
constituency where the intention is to expand the opportunity space to allow 
identifying the best alternatives, the decision still remains with the political 
level. And the processes and decisions at this level are not always rational, as 
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Figure 6 Categories of projects within the opportunity space. 
Source: Whist and Christensen (2011). 


illustrated in Fig. 6. The two dimensions of rationally derived and politically 
feasible span four categories: 


e The win/win projects score well on both dimensions and “must be im- 
plemented” (hydro power plant with no environmental downsides) 

e Rational projects, but which are not politically mature, where a quality- 
at-entry approach, such as the Norwegian Ministry of Finance QA re- 
gime can aid in the decision process to get these promoted (close down 
nuclear power plants) 

e Politically acceptable, even desired, but poorly conceived projects, these 
should be stopped, and the QA regime can help clarify the financial re- 
alities and thus kill such initiatives (Olympic games in a small country) 

e The lose/lose projects have no support in either direction and should 
never go further (private exploration of space). 


A separate case study of 23 major public investment projects (Whist and 
Christensen, 2011) went deeply into how the analytical and political pro- 
cesses interacted during the frontend phase, in order to understand how this 
affected the outcome of the projects. It was found that the majority of pro- 
jects started out with a predetermined solution. In about half the cases an 
unambiguous problem analysis was nevertheless carried out, and in one third 
of the cases new problems were introduced during the front end phase, Fig. 
7. The result was that two third of the projects were initiated with the same 
conceptual solution as the initial one, while in one third of the cases the con- 
ceptual solution was a different or changed substantially. Ten of the projects 
were considered relevant in relation to needs in society. Nine of these had a 


Appendix 219 


Predetermined Conceptual solution 
solution: 90 % same as the initial 60% 


Unambiguous 
problem analysis: 


50% Conceptual solution 


different from the initial: 
40 % 


No predetermined 
solution: 10 % 


New problems 
introduced later 30% 


Figure 7 Path dependency in defining and agreeing on conceptual solutions up front. 
Source: Authors. 


comprehensive problems analysis up front, and the Government had been a 
central actor in seven of them, while only in two of the thirteen projects were 
considered less relevant. 

These studies, and the examples mentioned, first and foremost illustrate 
the unpredictability of the political system in a mature democracy; a well 
developed, rational decision basis is no guarantee for a rational choice of con- 
cept. It was concluded that a scheme with external quality assurance of the 
decision basis provided to the political level had proved to have some positive 
effect in terms of helping make some choices more rational. 

While the analytical part of the decision-making processes overall was 
rather weak, the participation of and control with the participating actors was 
considerable in these projects. From experience we know that a bad starting 
point may be adjusted through a successful decision-making process, even 
when the original idea was quite wrong. We also know that in many cases 
this does not happen. 

This study demonstrated that there are many hurdles for any project. Dem- 
ocratic decision-making processes, particularly those which take long time, 
are complex and difficult to predict, and many will claim that this is a neces- 
sary part of democracy. If this is taken as a premise, the study suggested that 
the biggest potential for improvement lies in strengthening the analytical 
process. What would seem to be a reasonable compromise in front end anal- 
ysis and quality assurance of major projects would be that the first step should 
be to identify and eliminate the worst alternatives. These are low hanging 
fruits and proper action can give a high reward with little effort. The next 
step should be to seek for good alternative concepts, but within reasonable 
limits, and not necessarily crave for the best, since the case will nevertheless 
be handed over to decision makers to conclude. 
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7 The paradox of strategic alignment: strategy and 
alignment of objectives are highlighted as essential 
concerns, but in most cases the internal logic of 
causalities and the probabilities of realization are 
erroneous 


Alignment of objectives is the exercise to define the basic logical structure 
outlining the project by following the causal link from the basic needs of 
users and society, through defined goals to the delivery of project results 
(outputs), their outcome (effects) and long-term benefits after the project is 
terminated (purpose). This needs to be done before starting significant work 
on a project or programme. Unfortunately, this is not always done and can 
result in significant underperformance compared to expectations (Cooke- 
Davies, 2011). 

Any large projects, and particularly major public investment projects, are 
initiated in order to produce benefits for their owners (society). Many au- 
thors have studied success factors and predictors of failure, notably Morris and 
Hough (1987), Pinto and Slevin (1988), Miller and Lessard (2000), Flyvbjerg 
et al. (2003), and Hopkinson (2007). The available literature provides several 
different answers to why things go wrong and what could bring success in 
projects. Earlier literature tended to focus narrowly on the outputs in terms 
of cost, time and quality, whereas more recent literature may offer a wider 
perspective. For example, Morris and Jamieson (2005) study the processes, 
practices and people issues involved in moving from corporate strategy to 
projects. Their results are promising, though only based on four case studies. 
A common feature is that when projects’ strategic success is low, the problem 
possibly lies in the early phases of the project and at the governance level (the 
owner perspective). 

Some studies on international development projects have provided insight 
in this area. An analysis of a large sample of such projects concluded that most 
of the uncertainties affecting these projects were internal and not contextual, 
for a large part associated with aspects of management and the fundamental 
project design (Samset and Haavaldsen, 1998). Consequently, the suggestion 
was that most of the problems ought to be met early, i.e. in the pre-study 
phase. 

Youker (1999) concluded that the lack of shared objectives and agreement 
on the objectives of a project was one of the biggest problems facing interna- 
tional development projects. A study of alignment of objectives in develop- 
ment projects based on a sample of 30 international aid projects, concluded 
that most of the projects had design faults at all levels, and no projects were 
without faults. Typical problems identified were insufficient resources, and 
too many and unrealistically ambitious goals (Samset, 2006). 

The same analysis was repeated on a sample of 17 large public investment 
projects in Norway (Andersen et al., 2014). A project strategy will always 
be a hierarchy of goals that are interlinked in cause-and-effect chains that 
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Figure 8 Assessment of the goals in the sample of projects in terms of location in the 
goal hierarchy and their level of ambition (Samset et al., 2014a, 2014b). 


illustrate the ambition levels for a project, as well as their realism. Objective- 
swere analysed in terms of their internal causality, and ambition. Complex 
statements were broken down in several single objectives. 

The study found that in most of the Norwegian projects the goals are 
consistent with the needs, but there were shortcomings when it comes to 
achieving reasonable levels of clarity and ambition, as shown in Fig. 8. For in- 
stance, when a project to acquire defence equipment presents “stability within 
the international legal system” as a societal goal and a limited road construction 
project expects to result in “increased settlement”, we intuitively understand 
that the distance between cause and effect is too large and that the goals are 
too ambitious for the given project. 

Fig. 8 compiles the findings from the study and breaks down the percent- 
ages of goals across the different goal levels. Of the total 152 goals presented 
by the 17 projects, by far most of these were defined as project outcomes with 
the majority of the remaining goals being societal goals. About a quarter of 
the project outcomes were in reality societal goals, while two thirds of the 
presented project outcomes actually were project outcomes. Also, a small 
portion of the social goals are completely unrealistic, while a small set of the 
project outcomes were in reality project outputs, 1.e., specifying aspects of the 
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project’s deliverables. In total, none of the projects avoided erroneous defini- 
tions of goals, but they performed better than the international development 
studies mentioned above. But clarity seemed to be the largest problem. Five 
of the projects had in reality no societal goals whatsoever, while others had 
too many. One project had as many as seven societal goals. In such a case, 
the strategy is of little help to focus the efforts and clarify the purpose of the 
project. Regarding project outcomes, the majority of projects had 3-9 project 
outcomes, two projects even more than 10. 

The purpose of formulating an objective is principally to clarify the di- 
rection for that which is sought. The scope also needs to be stated so one 
may know when an objective is attained. Multiple objectives may confuse 
if they all don’t point in the same direction. This is particularly evident if 
the objectives also conflict with each other. Objectives should give rise to 
common understanding among and motivation of all parties involved in or 
affected by a project. On one hand, this means that objectives should be un- 
ambiguous and realistic. On the other hand, to motivate, they also have to be 
well founded, to the degree that they are accepted. Moreover, the objectives 
should limit the enterprise or the strategy. This means that the resources al- 
located and the results anticipated should correspond. 

In looking at customary practice in planning projects, the threshold for 
improvement seemingly is very low and the possibilities of marked improve- 
ment accordingly are great. Regardless, practice indicates a need for more 
concise formulation of objectives in the front end phases of projects, at any 
rate to establish common understanding of where a project is going and how 
it will get there. 


8 The cost estimation paradox: the focus is on the final 
cost estimate (the budget), while early cost estimates 
are overlooked 


We have already discussed how planners devote less attention to identifying 
the best conceptual solution than to improving tactical project success. This 
is understandable to some extent because planners find it easier to relate to 
tangible and quantified success criteria such as cost and time, than to mul- 
tidimensional and qualitative assessments of societal benefits. However, the 
investment cost is tangible and concrete, and crucial both to the choice of 
concept and to tactical success. Although cost uncertainty is higher in the 
early stages, it too is tangible and manageable (e.g. Austeng et al., 2005). 
Planners should therefore be strongly committed to establishing a rough but 
realistic cost estimate in the early phase, for comparison with project benefits. 

Under the auspices of the Concept research programme a study of cost 
estimates in projects’ initial phase has been conducted (Welde et al., 2014). 
The study explored a sample of 12 projects to determine the basis for and 
how the first cost estimates came about and developed during the whole 
period from the first initiative that was taken until the project was approved 
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The first estimate as compared with the final cost (Per cent) 


Figure 9 The earliest cost estimate as a percentage of the final cost, for 12 Norwegian 
projects. 
Source: Welde et al. (2014). 


by Parliament. As shown in Fig. 9 the first cost estimate in all 12 cases was 
far below of what was ultimately approved as the projects’ final budget. The 
increase in cost estimates during the front-end phase ranged from +70% to 
almost +1300%, with an average of +650%. By comparison, the cost increase 
during the implementation phase was much less, and some projects were even 
completed below budget; the variation ranged from —19% to +186%. 

The study is a first probe into the matter of early cost estimation. More re- 
search is needed to determine the extent of the problem and its implications. 
However, it indicates that initial underestimation may be significant and re- 
sult in the approval of projects that otherwise should have been rejected in the 
early stages. The authors considered it likely that at least 5 of the 12 projects 
would have been screened out at an early stage if the first estimate had been 
at a realistic level as compared with what was the final cost. The question is 
of course hypothetical, but there is no doubt that underestimation of costs 
at an early stage can have dramatic implications for project selection and is 
probably a far more severe problem than cost overruns in the implementation 
phase. Hence, it is clearly a paradox that so little attention is devoted to the 
initial estimate. 
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The report discusses possible reasons for the substantial underestimation 
in early phases. An often used distinction is made between political, techni- 
cal, and cognitive reasons (e.g. Flyvbjerg, 2005). It may be very difficult to 
prove that the cause is political, but in several of the projects there were clear 
indications that the first estimate was deliberately low in order to increase 
the chance of the project idea being considered. This corresponds well with 
other studies that have attempted to prove that costs are underestimated de- 
liberately to make the projects appear more attractive (e.g. Flyvbjerg, 2007; 
Mackie and Preston, 1998; Wachs, 1987; Welde et al., 2014). Wachs (1989) 
discusses how the most effective planner is sometimes the one who can cloak 
advocacy in the guise of scientific or technical rationality. In other cases the 
total investment was split between several projects that would have to be 
approved separately in sequence. However, in the most of the projects there 
were also different cognitive reasons why costs had been underestimated up 
front. Over-optimism is a well-known phenomenon in cognitive research 
literature, see e.g. Kahneman and Tversky (1979). Further, the study discusses 
measures to reduce the problem of early underestimation, such as systematic 
recording of early cost estimates, the use of reference projects, of stochastic 
costestimation techniques, increased provisions for uncertainty to account for 
possible scope changes, and third party review at an early stage. 


9 The paradox of disregarded analyses of costs and 
benefits: detailed estimation of cost and benefits is 
commonly done up front, but disregarded by decision- 
makers, who tend to emphasize other aspects 


A substantial amount of resources is devoted in major investment projects 
to establish a decision basis. Detailed Cost-Benefit Analyses are often per- 
formed, and complex models are developed to simulate traffic volumes and 
other inputs to these analyses. However, there are indications that decision- 
makers have little confidence in Cost Benefit Analysis in Norway. 

The transport sector is a special case. In this sector there is a long tradition 
of using Cost Benefit Analysis. A recent study conducted by the Concept 
research programme, Welde et al. (2013) studied the significance of Cost- 
Benefit Analysis in the final prioritization of road projects in Norway and 
Sweden, where the approaches to such analyses in the two countries are very 
similar and unit prices are of the same magnitudes. The study revealed that 
the Cost-Benefit ratio had no significant impact on the selection of projects 
in Norway. On the contrary, many unprofitable projects were realized, such 
as spectacular tunnels and bridges in sparsely populated areas. By contrast, 
in Sweden, the results of the Cost-Benefit Analyses had somewhat more 
influence on the selection of road projects. Clearly, in the case of Norway 
there must have been other factors thatweremore important but that were not 
included in the analyses. 

One explanation for low confidence in the Cost-Benefit Analyses could be 
weaknesses and shortcomings in the methodology, see e.g. Næss (2006, 2012). 
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O Coastal express line 25 20 43 65 37 1 30 19 40 
I Sea transport and fishery 50 45 32 15 22 61 9 13 17 


Figure 10 Percentage of the total monetized benefits in nine different Cost-Benefit 
Analyses of the Stad shipping tunnel, showing how much weight was 
placed on the various components. 

Source: Kvalheim (2014). 


The trend is however that more and more effects are included in the analysis, 
and the empirical basis for estimating realistic values is improving. See for 
example Vickerman (2008) on the inclusion of so-called wider economic 
benefits from transport infrastructure projects. Another explanation for low 
confidence could be strategic use of analyses to promote a desired result. One 
study, by Kvalheim (2014), examined a special case where nine Cost-Benefit 
Analyses had been made of one project, a shipping tunnel on the west coast of 
Norway. This study found a remarkable lack of consistency between analyses. 
The analyses were performed between 1990 and 2012, and the Cost-Benefit 
ratio varied from 0.2 (highly unprofitable) to almost 1.0, and even exceeded 
2.0 (highly profitable) in an ‘optimistic calculation’ provided in one of the 
reports. The analysis reporting the most positive number was funded by local 
stakeholders, with no financial obligations. An interesting finding was that 
the relative weight put on different benefit components varied noticeably, as 
shown in Fig. 10. This underscores the credibility of such studies. By 2014, 
the tunnel project had still not been approved for funding or finally rejected. 

Not all effects of an investment project may be quantified and expressed 
in monetary terms. Nevertheless, if they are relevant to the decision they 
should be systematically reviewed as much as the net present value. Norway is 
often regarded to be at the forefront internationally when it comes to includ- 
ing non-monetized impacts in Cost Benefit Analyses. However Bull-Berg 
et al. (2014) reviewed a practice regarding nonmonetized impacts in more 
than 100 economic analyses in Norway. With a few important exceptions, 
their findings are rather discouraging. The section presenting non-monetized 
impacts in the economic analysis is characteristically short, and not based 
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on transparent methodology and well-documented processes. The study 
concluded that there is substantial potential for improvement and a need for 
guidance. 

The paradox in this case is that so much effort is devoted to the calculation 
of a net present value that decision-makers may not find useful or credible. 
Clearly, planners should focus more on non-monetized impacts in economic 
analyses, as well as other complementary analyses such as cost-effectiveness 
analysis, impact evaluation, and multi-target criteria analysis. In addition, 
competence requirements are crucial to ensure high-quality analyses. 

The above situation is mirrored in the World Bank, which made wide 
use of Cost Benefit Analyses for decades to demonstrate its reputation as a 
knowledge bank and its commitment to measuring results and ensuring ac- 
countability to taxpayers. However, according to the World Bank (2010), the 
percentage of projects justified by a Cost Benefit Analyses has been declining, 
and the Cost-Benefit ratio is now rarely mentioned in policy documents. 
These results are explained by a decline in adherence to standards as well as 
increased difficulty in applying Cost Benefit Analyses in new sectors where 
traditionally it has not been applied and where benefits can hardly be quan- 
tified. The situation is that economic assessments are not performed at all. 
The World Bank concludes that there is a need to recognize the difficulties 
in quantifying benefits, but at the same time quality, rigour, and objectivity 
must be ensured because poor data and poor analyses are misinforming and 
do not lead to improved results. 


10 The paradox of “predict and provide”: the tendency 
is to choose a “‘predict-and-provide”’ strategy rather 
than explore alternative solutions 


Different perspectives can be taken when evaluating the need for an invest- 
ment project. As discussed by Næss (2005), public planners tend to use a 
predict-and-provide approach. When confronted with capacity problems, 
the planners, who are often engineers, almost always recommend increased 
capacity based on estimates of future demand. However, unsurprisingly, 
there is often excess demand for public services and infrastructure offered 
free-of-charge to citizens. The need should not be defined narrowly as a need 
to increase capacity but rather as a need to solve the congestion problem. The 
latter allows for a variety of measures, including demand regulation, con- 
gestion pricing, and legal and informative measures, most of which are far 
cheaper than a construction project to expand capacity. 

Our suggestion that needs should be considered in a broader perspective is 
supported by Odhage (2012), who studied early project planning in Swedish 
road projects. He found that the planners were never truly interested in find- 
ing and developing measures that would reduce the need for transport. This 
is obviously an example of path dependence, and Odhage asked the timely 
question ‘Can one expect anything different from a process that is run by the 
transport administration and concerns transport issues?’ 
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Traffic (AADT) 
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With traffic reducing 
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Political objective 


time 
Today 


Figure 11 Traffic development in different scenarios, illustrating that the need for 
an infrastructure project follows from the assumption that capacity should 
adapt to demand. 


Source: Authors. 


Further, in many cases there are political goals for a development that 
is quite the opposite of a predict-and-provide strategy. Næss (2005) distin- 
guishes between (1) needs defined by national-level political objectives, (2) 
market-based needs as measured by demand or willingness-to-pay, and (3) 
the needs of different stakeholder groups. As noted, public planners tend 
to narrow down the identification of needs to the second demand, while 
ignoring the broader spectrum of needs, and even political goals to reverse 
the demand trend. A country with high ambitions to reduce the emission of 
greenhouse gases will view increased traffic (i.e. growing demands for roads) 
as a problem.” Similarly, in the university system, a purely demand-based 
approach probably would not necessarily lead to a distribution of graduates in 
line with society’s need for expertise in different disciplines. 

The paradox in this case occurs when needs and benefits assessments in 
public infrastructure projects are decoupled from overriding political priori- 
ties and goals, possibly because such overriding societal goals are conflicting 
and multidimensional. The result of this is that issues such as scaling and ca- 
pacity of infrastructure projects, highly political choices, are left to planners, 
who (1) have a tendency to define the problem narrowly as absence of capac- 
ity, and (ii) use readily available estimates of demand as a reference for adjust- 
ing capacity. There is obviously a need for project owners (the government) 
to clarify what needs should be taken as a starting point for planners, and 
to express them as clear objectives for the project. Only if the development 
given by trend extrapolation is a clearly desired one can the predict-and- 
provide strategy be readily used in individual projects, as illustrated in Fig. 11. 


3 Ina separate study, Hagen (2010) discusses economic measures as accounting for external 
effects on the environment. 
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11 The paradox of perverse incentives: public 
investments with no financial obligations for the 
target group may cause perverse incentives and result 
in counterproductive projects 


The state often appears as a generous donor on behalf of taxpayers when 
financing projects that benefit specific groups or geographical regions. Such 
projects may be initiated either by the beneficiaries themselves or by the state 
out of pure altruism. There are indications that such projects often prove 
unsuccessful in strategic terms, and we should not be surprised by this. When 
a project does not entail financial obligations for recipients, there is no in- 
centive to opt for the most socially beneficial or cost-effective alternative. 
Different actors may have a vested interest in certain projects being chosen. 

The term perverse incentives refers to the situation where one or more 
actors are motivated to make choices resulting in a project that is a complete 
failure seen in retrospect. The theoretical basis is the principal-agent theory 
(e.g. Jensen and Meckling, 1976; Laffont and Martimort, 2002). There is a 
huge amount of literature on incentive problems in general, but less in rela- 
tion to state-funded investment projects. A pivotal study in the field has been 
published by Ostrom et al. (2001), who demonstrate serious problems with 
perverse incentives in Swedish-funded aid projects that resulted in the waste 
of public funds and adverse side effects such as corruption. 

Norway is a special case because the local government is financially weak 
and dependent on the state to finance local infrastructure. The State is rich 
due to revenues from the exploitation of petroleum resources. Whist and 
Christensen (2011) demonstrate how the early phase of state-funded invest- 
ment projects in Norway is often characterized by ‘local rationality’ and 
complex coalitions. Samset et al. (2014a, 2014b) explored the phenomenon 
of perverse incentives in nine Norwegian state-funded projects, to illustrate 


State-funded 
Total cost without Operational} Tactical Strategic 
(mill. NOK) | liabilities for success success success 
beneficiaries 


Project ype of project 


Hvaler-tunnelen Subsea road tunnel 200 No Yes Limited Limited 


Linesgya i 250 Minimal No No No 
St. Olavs Hospital i Minimal No Limited Limited 


Turkana Fisheries (Kenya) |Development aid Yes No No No 


(OL Lillehammer Sports event Yes Limited Yes No 


E16 Lærdalstunnelen Road tunnel Yes Yes Limited No 


Lofast Subsea road tunnel Yes Yes Limited No 


Rock city Cultural building Yes Limited No No 


Stad skipstunnel Shipping tunnel Yes n/a n/a Not likel 


Figure 12 Selected findings from Samset et al. (2014a, 2014b), one aid project and 
eight Norwegian state-funded investment projects without liabilities for 
the target group. 
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how perverse incentives might occur, what the causes and consequences 
might be, and what could be done to avoid them. One aid project served as a 
reference case to demonstrate how wrong things can go. The study revealed 
that half of the Norwegian projects scored very poorly, particularly in a stra- 
tegic perspective (Fig. 12). Some of these projects clearly would not have 
been prioritized had the recipient been required to contribute to the funding. 
Several projects were classed as supersized because they were ‘freeof- charge’. 
Moreover, Samset et al. (2014a, 2014b) found that costs were being underes- 
timated and benefits overestimated in advance. 

The problem of perverse incentives is twofold: (1) actors who act out of 
self-interest, and (2) a financing party that fails to reveal that. Measures to 
solve or mitigate the problem should therefore also be twofold: (1) aligning 
recipients’ objectives with national objectives, through requirements such as 
co-financing and local risk taking, and (2) reducing the information asym- 
metry by introducing, for example, by information control, external review, 
and public hearings. The Norwegian quality assurance regime is thus a meas- 
ure that is expected to reduce the problem of perverse incentives. 


12 The paradox of myopic decisions: long-term viability 
is the intention but the planning horizon is too short, 
resulting in sub-optimal choices that one will regret 
later 


Probably the most crucial strategic success criterion for an investment project 
is that it is viable and sustainable, i.e. that project net benefits are likely to 
continue in the long run (OECD, 2000). 

Viability can only be determined in the very long run. Samset (2012) stud- 
ied 10 projects from history, and found that only a few were still consid- 
ered highly successful and thus viable more than 100 years after completion, 
whereas others had been closed down after a short time. Needs and priorities 
in society may change over the years, and therefore a project’s viability is 
contingent upon its ability to adapt to changing needs. Ironically, one of the 
most viable projects in the study was the Eiffel Tower, which was built for no 
other purpose than to be an exhibition object to showcase France as a leader 
in science and technology, but which later became one of the greatest tourist 
attractions in the world. 

Since viability can only be determined in the long run, an assessment of 
viability ex ante must have a long-term perspective and the planner must be 
able to think creatively about possible future scenarios. It is not sufficient that 
the project is feasible and relevant on the opening day; planners must con- 
sider whether it will continue to be so throughout its lifetime. Lædre et al. 
(2012) studied 24 appraisal reports of major public projects from the periods 
2005-2011 with respect to their assessments of viability. The results were 
rather disappointing: needs and benefits were most often assessed in a short- 
sighted and static perspective; trends were extrapolated without discussing 
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Project 1 


+ 


Net benefits = 
Benefits - costs 0 i 


+ 


Net benefits = 
Benefits - costs 0 


Figure 13 Illustration of myopic decisions. Two projects with identical investment 
cost have different net benefit flows throughout their life-time. In a long- 
term perspective it is clear that project 2 is more viable, but a myopic plan- 
ner would emphasize short-term effects and choose project 1. For example, 
investments in preparedness and prevention capacities are often very low, 
something that one regrets later when a disaster strikes. 

Source: Authors. 


alternative scenarios; most attention was devoted to tangible effects, ignoring 
non-monetized impacts; and significant risk factors, such as political risk, 
were not identified and discussed. Such practice may lead to myopic deci- 
sions, which we are likely to regret in the future, as illustrated in Fig. 13. 

However, Lædre et al. (2012) also noted that no single analytical tool is 
able to comprehend all aspects of a project’s viability ex ante. In particular, a 
Cost-Benefit Analysis, although intending to capture all economic impacts 
of a project, cannot provide sufficient analysis of viability, one important 
reason being the use of a discount rate. Therefore, in order to assess viability 
properly, several complementary tools combining quantitative and qualitative 
approaches are necessary. A separate study by Hagen (2011) goes further into 
the question of how the Cost-Benefit Analysis, through the use of a discount 
rate, leads to short-termism and neglect of future generations. However, 
Hagen also shows that it may be appropriate to use a decreasing discount rate 
over time. This would in fact increase the planning horizon and thus mitigate 
the problem. 

The paradox in this case is that the emphasis on viability as a success cri- 
terion is far from reflected in project appraisals. Projects that are meant to 
last for decades and sometimes centuries may have significant impact on 
economic, environmental, and social development, yet they are still as- 
sessed in a short-term and static perspective. Lædre et al. (2012) offer some 
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recommendations for how to obtain a broader and more long-term perspec- 
tive in project appraisals. They involve shifting the analyst’s attention away 
from detailed estimations of investment cost to estimating future benefit 
flows and corresponding risk. Undoubtedly, evaluating a project’s viability 
ex ante can be challenging, but the alternative of finding out about its unvi- 
ability too late is worse. 


13 Discussion 


Governance regimes for major investment projects comprise the processes 
and systems that need to be in place on behalf of the financing party to ensure 
successful investments. What happens during the front end phase is essential. 
Peter Morris (2011:7) writes that “It is evident from an extensive amount of 
research that management of the front-end definitional stages of projects is 
of overwhelming importance to their ultimate outcome yet we have little 
empirical data to suggest how best management competencies here should 
be improved.” 

Project governance has only recently become an issue in the project man- 
agement community. In order to move forward in this field we have to find 
answers to what would be the optimal mix of regulations, economic means 
and information in improved governance regimes. What seems to be an issue 
for the project management community 1s to lift their perspective beyond the 
delivery of the project itself and onto the broader issues of the project’s utility 
and effects. It is obviously not only about the quality of analyses up front but 
also about decision processes. To arrive at the optimal conceptual solution 
based on rational analysis is of little worth if it is not the one chosen. 

The Concept programme did a pilot project on a sample of cases to il- 
lustrate this (Samset, 2008a, 2008b), which was followed up with a more in 
depth study to explore the quality and interaction between analysis and deci- 
sions during the front end phase (Whist and Christensen, 2011) and a broader 
followup of the pilot (Samset and Volden, 2013a, 2013b). The result is dis- 
played in Fig. 14, where the flaws for the individual projects are plotted with 
“X”. The summary row at the bottom are marked to signify whether they 
are considered relevant as seen in relation to needs and priorities in society 
(white colour) or not (black). Each project is represented with one column. 
The columns are sorted from left to right according to the observed number 
of flaws. The resulting pattern suggests that the least relevant projects have a 
lot of flaws in their analytic and decision making processes (between five and 
ten). The ones that are regarded relevant on the other hand have much less 
flaws (between one and four). 

The studies concluded that there is a strong tendency to choose the initial 
concept and stick to it, almost regardless of how bad it is. Also, there is an 
overwhelming inertia. Once the train has been set in motion — it is always 
impossible to stop. This goes a long way to explain the red projects on the 
left hand side. Further there is a third common tendency, that incremental 
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Project number sum 


Analysis 12 14) 
Alternative concepts have not been scrutinized 
Strategic underestimation of expected costs 


Inadequate/limited analysis of problems and needs 
Lack of realistic objectives and justification 

Tactical splitting up and sequencisng of project 
Predictable surprises not taken into account 
Decisions 


Disagreement regarding objectives and justification 
Expert advice overruled by political preferences 


Long lasting front end phase with shifting priorities 
Repeated playoff in political decision process 
Perverse incentives - benefits without liability 
Political horse-trading between competing parties 
Sum 


Relevance of the project 


Figure 14 There is a consistent tendency that projects that are considered relevant 
have less flaws in the analysis and decision making processes up front 


(Samset, 2008a, 2008b). 


improvements of an inferior solution are preferred rather than fundamental 
change. 

On the other hand experience also suggests that the opportunity space is 
usually larger than envisioned — and it is often largely unexplored. What was 
evident, however, was that the green projects seemed to have been exposed 
to more vigorous analyses and decision processes that were less affected by 
disagreements, political preferences, lengthy processes and repeated playoffs 
in the political decision processes. 


14 Conclusions 


This paper reports from several in-depth case studies of major public pro- 
jects, and identifies a number of paradoxes that could guide further research. 
In various ways the paradoxes point to two types of problems, i) problems 
of efficiency in terms of delays and cost overrun, and 11) more fundamental 
problems that have to do with the project’s strategic success (choosing the 
wrong concept). Project management as a discipline should be concerned 
with both problems. To quote Peter Morris: “The discipline needs to be less 
inward looking: more relevant, not just to the sponsor’s needs but to society’s 
challenges in general. We can foresee several changes in the years ahead in 
the ways projects and programs will be managed, but the obvious immediate 
needs are to focus more on improving sponsor value and on shaping the con- 
text in which projects and programs are formed and implemented” (Morris, 
2013:23). 

Many of the problems facing major public investment projects can be in- 
terpreted in terms of deficiencies in the analytic or the political processes 
preceding the final decision to go ahead, and the complexity and uncertainties 
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affecting these processes. In particular, the fundamental problems with stra- 
tegic success could typically be traced back to deficiencies in the earliest 
preparatory phases of the project. The role of the front-end phase in ensuring 
project success is therefore crucial, as highlighted in the literature (Merrow, 
2011; Morris, 2013). 

Project governance is the processes, systems, and regulations that the fi- 
nancing party must have in place to ensure that projects are successful, stra- 
tegically as well as tactically. Many organisations have introduced stage-gate 
phase models, also the Norwegian Ministry of Finance, who introduced a 
QA scheme to ensure the best choice of concept (QA1) and efficient project 
implementation (QA2) in year 2000. Our research indicates that QA2 has 
already led to improved cost control. It is still too early to conclude that QA1 
has improved the choice of conceptual solutions and projects’ strategic suc- 
cess, but there is evidence to suggest that an independent review of the pro- 
ject appraisal documents at a very early stage has a positive effect. There are 
many fundamental challenges that will have to be dealt with, such as tactical 
budgeting in local communities and responsible agencies at various levels, 
which is done in order to increase the chance to obtain government funding 
for a project. Another challenge is to ensure a transparent and democratic 
process and avoid adverse effects of stakeholder’s involvement and political 
bargaining. But also to make the process predictable is a major challenge. The 
QA regime attempts to remedy these problems. 

One salient conclusion from the research is that ex post evaluation should 
be an essential element in any project governance scheme. When a project 
succeeds at all levels, it should be imperative to ask what was done right. Cor- 
respondingly, one should learn from mistakes. However, experience shows 
that the use of evaluations for learning purposes is limited, and this is particu- 
larly true in the public sector (Samset and Christensen, 2012). The tendency 
is to look ahead with the concern of how to spend next year’s budget, rather 
that look in the rear mirror to learn from experience. 

As a lead part of the current trailing research on Norwegian public pro- 
jects, the Concept research programme has since its inception been concerned 
with project evaluation and evaluation methodology as evidenced in several 
studies, including those by Olsson (2005), Andersen et al. (2007), and Volden 
and Samset (2013). The latter is a summing-up of four pilot evaluations of 
so called QA projects. It recommends that systematic ex post evaluations of 
public investment projects should be carried out to learn from experience, 
not least how they perform in a strategic perspective, with the aim to im- 
prove public investment projects in the future. Under the auspices of the pro- 
gramme therefore, a number of the major investment projects are now being 
evaluated, and this will continue in the years to come. Fig. 15 shows some 
main results for the first nine projects. 

Clearly, projects may fail even when formal rules for planning and deci- 
sion making have been adhered to. Democratic decision-making processes, 
particularly the long lasting ones, are complex and the outcome difficult to 
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Project Efficiency Effectiveness 
(tactical project success) (strategic project success) 
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Sandvika-Asker inter-city rail line (x ë ă O Lk # __) 
Momarken-Sekkelsten, section of a highway (KK) (À J 


ISkjold class missile torpedo vessels 
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[Svalbard Science Center 


Figure 15 Main results from ex post evaluations of nine Norwegian investment pro 
jects (three stars = high success, two = medium success, one = unsuccessful). 
For more detailed results, see the evaluation reports, available on www. 
ntnu.no/concept. 


predict. Many will claim that this is a necessary part of democracy. If this is 
taken as a premise, one could conclude that the biggest potential for improve- 
ment lies in strengthening the analytical process, as well as making decision 
processes transparent. 
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